{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "1.为什么Bagging能改进模型性能？\n",
    "答：bagging是基于弱模型（偏差小，方差大）。偏差小准确率高，方差大防过拟合能力弱。bagging中整体模型的期望近似于基模型期望，所以整体模型\n",
    "偏差和方差近似于基模型的偏差和方差。随着基模型的数量增多，整体模型方差减小，防过拟合能力增大，模型性能增强。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "2.随机森林中的随机体现到哪些方面？\n",
    "答：1.随机选择一部分特征\n",
    "    2.随机选择一部分样本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "3.GBDT和随机森林都是基于决策树。这两种模型中决策树有什么不同？\n",
    "答：随机森林的决策树可以是分类树，也可以是回归树，组成GBDT的决策树只能是回归树"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "4.请简述LightGBM的训练速度为什么比XGBM快？\n",
    "答：XGBM采用的贪心算法，需要对训练特征进行排序，遍历整个训练数据集多次。如果把整个数据集装进内存则会限制训练数据集的大小，如果不装进内存，反复读写训练数据集\n",
    "又会消耗非常大的时间。\n",
    "LightGBM使用基于直方图的算法，将连续的特征值分桶装进离散的箱子，直方图作差不需要遍历所有数据，只需要遍历直方图的桶。\n",
    "其中针对采样的优化又提出了GOSS，取出梯度小的样本，针对稀疏的特征又又EFB合并特征。\n",
    "所以Light比XGBM快很多"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "5.采用tfidf特征，采用LightGBM（gbdt）完成Otto商品分类，尽可能将超参数调到最优，并用该模型对测试数据进行测试，提交Kaggle网站，提交排名。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "import matplotlib.pyplot as plt\n",
    "import _pickle as cPickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.081393</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.075886</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.231403</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.199730</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.011987</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.011668</td>\n",
       "      <td>0.105971</td>\n",
       "      <td>0.021681</td>\n",
       "      <td>0.080435</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.008244</td>\n",
       "      <td>0.022456</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.124622</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.145988</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Class_1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 95 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0   1      0.081393           0.0           0.0      0.000000      0.000000   \n",
       "1   2      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "2   3      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "3   4      0.011987           0.0           0.0      0.011668      0.105971   \n",
       "4   5      0.000000           0.0           0.0      0.000000      0.000000   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  ...  feat_85_tfidf  \\\n",
       "0      0.000000      0.000000      0.000000           0.0  ...       0.075886   \n",
       "1      0.000000      0.000000      0.231403           0.0  ...       0.000000   \n",
       "2      0.000000      0.000000      0.199730           0.0  ...       0.000000   \n",
       "3      0.021681      0.080435      0.000000           0.0  ...       0.000000   \n",
       "4      0.000000      0.000000      0.000000           0.0  ...       0.124622   \n",
       "\n",
       "   feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  feat_90_tfidf  \\\n",
       "0       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "1       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "2       0.000000       0.000000            0.0            0.0       0.000000   \n",
       "3       0.008244       0.022456            0.0            0.0       0.000000   \n",
       "4       0.000000       0.000000            0.0            0.0       0.145988   \n",
       "\n",
       "   feat_91_tfidf  feat_92_tfidf  feat_93_tfidf   target  \n",
       "0            0.0            0.0            0.0  Class_1  \n",
       "1            0.0            0.0            0.0  Class_1  \n",
       "2            0.0            0.0            0.0  Class_1  \n",
       "3            0.0            0.0            0.0  Class_1  \n",
       "4            0.0            0.0            0.0  Class_1  \n",
       "\n",
       "[5 rows x 95 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv('Otto/train_tfidf.csv')\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LightGBM不支持字符串特征输入\n",
    "y_train = train['target']\n",
    "y_train = y_train.map(lambda s:s[6:])\n",
    "y_train = y_train.map(lambda s:int(s)-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = train.drop(['id','target'],axis = 1)\n",
    "#保存特征名后续可视化用\n",
    "feat_names = x_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 稀疏存储，模型训练更快\n",
    "from scipy.sparse import csr_matrix\n",
    "x_train = csr_matrix(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "# 分层交叉验证 越多越好\n",
    "kfold = StratifiedKFold(n_splits=3,shuffle=True,random_state=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. 调树的棵树n_estimators"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用内嵌lightgbm的交叉验证，对可连续的n_estimators进行快速交叉验证\n",
    "def get_n_estimators(params, x_train, y_train, early_stopping_rounds = 10) :\n",
    "    lgbm_params = params.copy()\n",
    "    lgbm_params['num_class'] = 9\n",
    "    lgbmtrain = lgbm.Dataset(x_train,y_train)\n",
    "    # num_boost_round为弱分类器的数目,设置的early_stoping_rounds即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    cv_result = lgbm.cv(lgbm_params,lgbmtrain,num_boost_round=MAX_ROUNDS,nfold=3,metrics='multi_logloss',early_stopping_rounds=early_stopping_rounds,seed=3)\n",
    "    print('best n_estimators: ',len(cv_result['multi_logloss-mean']))\n",
    "    print('best cv score: ',cv_result['multi_logloss-mean'][-1])\n",
    "    return len(cv_result['multi_logloss-mean'])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators:  396\n",
      "best cv score:  0.48664349894746767\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.1,\n",
    "         'num_leaves':60,\n",
    "         'max_depth':6,\n",
    "         'max_bin':127,\n",
    "         'subsample':0.7,\n",
    "         'bagging_freq':1,\n",
    "         'colsample_bytree':0.7}\n",
    "n_estimators = get_n_estimators(params,x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. num_leaves & max_depth =  7\n",
    "# 调叶子节点数目\n",
    "# num_leaves建议是70到80搜索区间50到80，值越大越容易过拟合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 8 candidates, totalling 24 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  10 tasks      | elapsed: 23.3min\n",
      "[Parallel(n_jobs=4)]: Done  22 out of  24 | elapsed: 48.2min remaining:  4.4min\n",
      "[Parallel(n_jobs=4)]: Done  24 out of  24 | elapsed: 48.7min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=3, random_state=3, shuffle=True),\n",
       "             error_score='raise-deprecating',\n",
       "             estimator=LGBMClassifier(bagging_freq=1, boosting_type='gbdt',\n",
       "                                      class_weight=None, colsample_bytree=0.7,\n",
       "                                      importance_type='split',\n",
       "                                      learning_rate=0.1, max_bin=127,\n",
       "                                      max_depth=7, min_child_samples=20,\n",
       "                                      min_child_weight=0.001,\n",
       "                                      min_split_gain=0.0, n_estimators=396,\n",
       "                                      n_jobs=4, num_class=9, num_leaves=31,\n",
       "                                      objective='multiclass', random_state=None,\n",
       "                                      reg_alpha=0.0, reg_lambda=0.0,\n",
       "                                      silent=False, subsample=0.7,\n",
       "                                      subsample_for_bin=200000,\n",
       "                                      subsample_freq=0),\n",
       "             iid='warn', n_jobs=4, param_grid={'num_leaves': range(50, 90, 5)},\n",
       "             pre_dispatch='2*n_jobs', refit=False, return_train_score=False,\n",
       "             scoring='neg_log_loss', verbose=5)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.1,\n",
    "         'max_bin':127,\n",
    "         'subsample':0.7,\n",
    "         'bagging_freq':1,\n",
    "         'colsample_bytree':0.7,\n",
    "         'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9}\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "num_leaves_s = range(50,90,5)\n",
    "tuned_parameters = dict(num_leaves = num_leaves_s)\n",
    "# refit 的意思是找到最佳参数后用全体数据训练一下模型，这里没必要所以false\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='neg_log_loss',verbose=5,refit=False)\n",
    "grid_search.fit(x_train,y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4860858162971072\n",
      "{'num_leaves': 75}\n"
     ]
    }
   ],
   "source": [
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'mean_fit_time': array([430.10992225, 455.57913264, 468.5963823 , 487.72173802,\n",
      "       470.39180374, 487.15664665, 496.9444859 , 493.8443683 ]), 'std_fit_time': array([ 8.83892726,  9.22386205, 10.57780438,  2.64989371,  1.6201838 ,\n",
      "       12.5383511 ,  1.99462713,  4.34430508]), 'mean_score_time': array([8.6141216 , 8.63789566, 8.738331  , 8.8232313 , 8.76941029,\n",
      "       8.5674425 , 8.82358925, 8.69983697]), 'std_score_time': array([0.10190127, 0.08943439, 0.09427491, 0.28129281, 0.05085586,\n",
      "       0.1735062 , 0.30475847, 0.80312619]), 'param_num_leaves': masked_array(data=[50, 55, 60, 65, 70, 75, 80, 85],\n",
      "             mask=[False, False, False, False, False, False, False, False],\n",
      "       fill_value='?',\n",
      "            dtype=object), 'params': [{'num_leaves': 50}, {'num_leaves': 55}, {'num_leaves': 60}, {'num_leaves': 65}, {'num_leaves': 70}, {'num_leaves': 75}, {'num_leaves': 80}, {'num_leaves': 85}], 'split0_test_score': array([-0.48893323, -0.48746234, -0.48647306, -0.48840163, -0.48849807,\n",
      "       -0.48720625, -0.4872757 , -0.48790217]), 'split1_test_score': array([-0.48402106, -0.48462756, -0.48560756, -0.48721705, -0.48654137,\n",
      "       -0.48417247, -0.48620783, -0.48530666]), 'split2_test_score': array([-0.48785282, -0.48880897, -0.48714104, -0.4886357 , -0.48746513,\n",
      "       -0.4868787 , -0.48961623, -0.48793755]), 'mean_test_score': array([-0.48693574, -0.48696625, -0.4864072 , -0.48808478, -0.48750156,\n",
      "       -0.48608582, -0.48769985, -0.48704879]), 'std_test_score': array([0.00210766, 0.00174269, 0.00062775, 0.00062098, 0.00079925,\n",
      "       0.00135953, 0.0014234 , 0.00123196]), 'rank_test_score': array([3, 4, 2, 8, 6, 1, 7, 5], dtype=int32)}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.cv_results_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEHCAYAAAB4POvAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeXxU9bn48c+TfSEkgSzsJkBYAsgWEUGoGxTbim0vWLR6e9ven7XVeq14q7a93la7Wlu7WVu7eO2tO+ottVahttYQFAn7TibseyYhCSF78vz+mBMcYjYgJ2cmed6vV17M+Z5zZp4ZAs+c7/d7nq+oKsYYY4ybIrwOwBhjTO9nycYYY4zrLNkYY4xxnSUbY4wxrrNkY4wxxnVRXgcQitLS0jQrK8vrMIwxJqysW7fOr6rpbe2zZNOGrKwsCgsLvQ7DGGPCiojsb2+fdaMZY4xxnSUbY4wxrrNkY4wxxnWWbIwxxrjOko0xxhjXWbIxxhjjOks2xhhjXGfJxhgX7DxWya5jp7wOw5iQYcnGGBd8+ZkNfOqJdzhcXuN1KMaEBEs2xnSzYxW1FJ2oory6gTueWU99Y7PXIRnjOUs2xnSz1cV+AG6/chQbDpTz8Os7PY7IGO9ZbTRjulmBr5TUhGiWzhtLVW0jv121lxnZA5g/YZDXoRnjGbuyMaYbqSqri/1cNmogERHC1z46nouHJXPPi5s4WFbtdXjGeMbVZCMiC0Rkl4j4ROS+Do5bJCIqInnOdrSIPCUiW0Rkh4jcH3TsV0Rkm4hsFZFnRSTOac8WkTUiUiQiz4tIjNMe62z7nP1Zbr5n07ft9Z/maEUts0alARAbFcljN01DwcZvTJ/mWrIRkUjgMeBaIBe4UURy2zguCbgTWBPUvBiIVdVJwHTgCyKSJSJDnWPzVHUiEAkscc75AfCoquYAJ4HPO+2fB06q6mjgUec4Y1xRUFwKwOzRaWfahg9I4IeLJrPpUAXffW2HV6EZ4yk3r2xmAD5V3aOq9cBzwPVtHPcQ8DBQG9SmQKKIRAHxQD1Q6eyLAuKdfQnAERER4CpgmXPMU8DHncfXO9s4+692jjem2632+RmSHEfWwISz2hdMHMTnZmfzP6v38dctRz2KzhjvuJlshgIHg7YPOW1niMhUYLiqvtrq3GXAaeAocAB4RFXLVPUw8IjTdhSoUNUVwECgXFUb23itM3E4+yuc443pVs3Nyjt7Spk9Oo22vs/cd+04Jg9P4avLNrO/9LQHERrjHTeTTVtXD3pmp0gEgW6tpW0cNwNoAoYA2cBSERkpIqkErlSynX2JInJzJ6/VYRxB8dwqIoUiUlhSUtL+uzKmHduPVlJe3XBWF1qwmKgIHrtpKhERwu3PrKe2oamHIzTGO24mm0PA8KDtYcCRoO0kYCLwlojsA2YCy51JAjcBr6tqg6qeAAqAPOAaYK+qlqhqA/AyMAvwAylO11rr1zoTh7M/GShrHayqPqGqeaqal57e5hLaxnRolS9wf82sUe1fOA9LTeBHiyez9XAl3/mLjd+YvsPNZLMWyHFmicUQGMhf3rJTVStUNU1Vs1Q1C3gXWKiqhQS6ya6SgEQCiWin0z5TRBKccZergR2qqsA/gEXO038G+JPzeLmzjbP/787xxnSrAp+fnIx+ZPSP6/C4a3IzuXXuSP733f38edORDo81prdwLdk44yN3AG8AO4AXVHWbiDwoIgs7Of0xoB+wlUDSelJVN6vqGgLjOeuBLU78Tzjn3AvcLSI+AmMyv3PafwcMdNrvBtqdgm3M+aprbGLtvrJ2u9Ba+88Pj2X6Ranc//IW9vpt/Mb0fmJf8j8oLy9PCwsLvQ7DhJF395Sy5Il3eeKW6V2uFHCkvIaP/iyfQcnxvPKlWcRFR7ocpTHuEpF1qprX1j6rIGBMN1jt8xMhcOnIrk90HJISz48/NYUdRyv51p+3uxidMd6zZGNMNygoLmXSsBSS46PP6bwrx2bwxStG8ex7B/jTxsMuRWeM9yzZGHOBquoa2XSwnNkdzELryNJ5Y5iRNYD7X96C70RVN0dnTGiwZGPMBXpvbymNzdrlyQGtRUVG8LMbpxIfHcntT6+npt7uvzG9jyUbYy5Qga+U2KgIpl+Uet7PMSg5jkc/NYXdJ07x38u3dmN0xoQGSzbGXKACn5+8rNQLnk02d0w6d1w5mhcKD/HSukPdFJ0xocGSjTEXwF9Vx85jp84sKXCh7rpmDDNHDuAb/7eVouOnuuU5jQkFlmyMuQCr21hS4EJERgg/WzKVxNgovvT0eqrrGzs/yZgwYMnGmAuw2ucnKS6KSUOTu+05M/rH8dMlU/CVVPGN/9uK3XhtegNLNsZcgIJiPzNHDiQyonuXSJo9Oo3/uDqHl9cf5sVCG78x4c+SjTHn6WBZNQfLas77/prOfPmqHC4fncZ//WkrO45Wdn6CMSHMko0x56nAWVKgu8ZrWouMEB791BT6x0dz+9Prqaqz8RsTvizZGHOeCopLyUiKZXRGP9deIz0plp/fOJV9paf52stbbPzGhC1LNsacB1XlnWI/s0YNbHMJ6O40c+RA7p43huWbjvDMewdcfS1j3GLJxpjzsOv4KfxV9cxyqQuttS9dMZq5Y9L51p+3s/VwRY+8pjHdyZKNMeehwNe999d0JiJCePSGyQxIiOGOZ9ZzqrahR17XmO7iarIRkQUisktEfCLS7gqZIrJIRFRE8pztaBF5SkS2iMgOEbnfaR8rIhuDfipF5C5n3/NB7ftEZKPTniUiNUH7fuXmezZ9w2qfn+y0RIamxPfYaw7sF8vPb5rKwZM13PeSjd+Y8BLl1hOLSCSB5Z3nAYeAtSKyXFW3tzouCbgTWBPUvBiIVdVJIpIAbBeRZ1V1FzAl6PkPA68AqOqngp7zR0BwX0Oxqk7p7vdo+qaGpmbe3VPKx6cO7fHXviRrAPfMH8sPXt/Jpe8O4F8vy+rxGIw5H25e2cwAfKq6R1XrgeeA69s47iHgYaA2qE2BRBGJAuKBeqD1jQZXE0gi+4MbJTBaewPwbLe8C2Na2XyonNP1TT3WhdbaF+aO5Mqx6Xz71R1sOWTjNyY8uJlshgIHg7YPOW1niMhUYLiqvtrq3GXAaeAocAB4RFXLWh2zhLYTyhzguKoWBbVli8gGEfmniMxpK1gRuVVECkWksKSkpLP3ZvqwAl8pInDZOSwB3Z0iIoQf3zCFtH4xfOmZdVTU2PiNCX1uJpu25oOe6WQWkQjgUWBpG8fNAJqAIUA2sFRERgadGwMsBF5s49wbOTsJHQVGqOpU4G7gGRHp/4HAVJ9Q1TxVzUtPT+/svZk+rMDnJ3dwf1ITYzyLITUxhp/fNI2j5bV8ddkmG78xIc/NZHMIGB60PQw4ErSdBEwE3hKRfcBMYLkzSeAm4HVVbVDVE0ABkBd07rXAelU9HvyCTrfbJ4HnW9pUtU5VS53H64BiYEy3vEPT59TUN7HhQLlnXWjBpl+Uyr0LxvHGtuM8WbDP63CM6ZCbyWYtkCMi2c6VyBJgectOVa1Q1TRVzVLVLOBdYKGqFhLoOrtKAhIJJKKdQc/d+uqlxTXATlU9U7lQRNKdyQQ4V0c5wJ7ufKOm71i7r4z6pmZmuVQP7Vz9+5xsrhmfyff+uoONB8u9DseYdrmWbFS1EbgDeAPYAbygqttE5EERWdjJ6Y8B/YCtBJLWk6q6GcCZnTYPeLmN89oax5kLbBaRTQTGgm5rY/zHmC4pKPYTHSnMyB7gdSgAiAg/WjyZjKQ4bn96PeXV9V6HZII0NjWzZk+pdXMCYh/CB+Xl5WlhYaHXYZgQdN3PVxEfHckLt13mdShn2XiwnMW/Ws2HxmTwm3+d7noJHdM1y9Yd4p4XN/G9T07ixhkjvA7HdSKyTlXz2tpnFQSM6aLy6nq2HqkIifGa1qYMT+H+a8fztx3H+W3+Xq/DMY5/7g7MbP32q9s5WFbtcTTesmRjTBe9u6cUVZg9OjTGa1r77OwsFkwYxA9e38m6/Se9DqfPa25WCnz+M+N7X122mebmvtuTZMnGmC5a5fOTGBPJ5OEpXofSJhHhB4suZnBKHF9+Zj0nT9v4jZe2H62k7HQ9i/OG8Y2P5fLOnlL+9939nZ/YS1myMaaLVvtKmZE9gOjI0P1nkxwfzS9vmo6/qp67X9jYp79Je+3tokAX2uzRaSy5ZDgfGpPO9/+6k33+0x5H5o3Q/VdjTAg5WlHDHv/pkByvaW3SsGS+8bHx/GNXCb9+22b5eyV/t5/xg/uTkRSHiPD9f5lEVKRwz4ubaOqDXwIs2RjTBS1LCswaFfrJBuCWmRfx0UmDeWTFLtbus5n+Pa26vpHC/WXMzXn/92VwcjzfvG4ChftP8vtVfW8ShyUbY7pgtc/PgMQYxg1K8jqULmn5Jj08NZ47nllPaVWd1yH1KWv2ltHQpFyec/aXk09OG8o14zP54Ypd+E5UeRSdNyzZGNMJVaWg2M9lowYSERE+968kxUXz2KencbK6ga+8sMnGb3pQ/m4/sVERXJJ19s2/IsJ3PzmRhJhIlr64icamZo8i7HmWbIzpRHHJaY5X1jE7TLrQgk0Yksx/X5fL27tL+OVbPq/D6TPyi0qYkT2AuOjID+zLSIrjoesnsulgeZ8aU7NkY0wnVhf7gdC9v6YzN80YwcLJQ/jxyt28U1zqdTi93rGKWopOVDE3p/3q8ddNHsJHJw3mJ3/bzc5jrZfq6p0s2RjTiQKfn6Ep8YwYkOB1KOcl0HUziayBidz53AZKTtn4jZvynSnPrcdrWnvo4xNJjo9m6QubaOgD3WmWbIzpQFOz8k5xKZePTgvremP9YqN47NPTqKxp4K7nN/TJqbc9Jb/IT1q/2E4nkwxIjOHbH5/EtiOV/OLvvb+L05KNMR3YdqSCytpGZoVpF1qw8YP78+D1EyjwlfLzvxd1foI5Z83Nyiqfn7k5XftysmDiID4+ZQiP/cPH1sO9e4lvSzbGdGCVLzBeEy7313TmhrzhfHLqUH76ZhEFznsz3aelRM2cMV3/ffnWwokMSIzh7hc2UtfY5GJ03rJkY0wHVvtKGZuZRHpSrNehdAsR4dufmMio9H78x3MbOFFZ63VIvUp+Uctkkq4nm+SEaH7wLxez+3gVP/lb773itGRjTDtqG5pYu6+sV3ShBUuIieKXn57G6bom7nxuQ5+618Nt+UUljBuUREZS3Dmdd+W4DG7IG8av/1nM+gO9s2K3q8lGRBaIyC4R8YnIfR0ct0hEVETynO1oEXlKRLaIyA4Rud9pHysiG4N+KkXkLmffN0XkcNC+jwQ9//1ODLtE5MNuvmfTe6w/cJK6xuawvL+mM2Myk3jo4xN5d08ZP32z936b7kk19U0U7jvJ3DHtT3nuyDc+lsug/nHc8+Imaht6X3eaa8lGRCIJLO98LZAL3CgiuW0clwTcCawJal4MxKrqJGA68AURyVLVXao6RVWnOO3VwCtB5z3asl9VX3OeP5fActETgAXAL53YjOnQal8pkRHCpSNDYwno7rZo+jAWTx/GL/7h421nkS9z/tbsLaW+qZk5nUx5bk//uGgeXjSZPSWneeSNXd0cnffcvLKZAfhUdY+q1gPPAde3cdxDwMNAcOexAokiEgXEA/VA6zufrgaKVbWzBSKuB55T1TpV3Qv4nNiM6VBBsZ+LhyWTFBftdSiuefD6iYzJSOKu5zdSZuvfXJD8Ij8xbZSoOReX56Rx88wR/K5gL+/t7V0FVN1MNkOBg0Hbh5y2M0RkKjBcVV9tde4y4DRwFDgAPKKqrT/5JcCzrdruEJHNIvJ7EUntahxOLLeKSKGIFJaU2Le8vu5UbQObD1X0yi60YPExkTyyeDJlp+t5fesxr8MJa/lFJVzaTomac3H/teMZlhrPfy7bRHV9YzdF5z03k01bk8zP3EkmIhHAo8DSNo6bATQBQ4BsYKmIjAw6NwZYCLwYdM7jwChgCoEk9aOuxHGmQfUJVc1T1bz09PPrczW9x5o9ZTQ1a6+bHNCWiUP7c9HABFZst2Rzvo5V1LL7eNV5d6EFS4yN4oeLJrO/tJrv/3VnN0QXGtxMNoeA4UHbw4AjQdtJwETgLRHZB8wEljuTBG4CXlfVBlU9ARQAeUHnXgusV9XjLQ2qelxVm1S1GfgN73eVdRaHMR9QUOwnLjqCaSNSOz84zIkI88ZnstpXSlVd7/km3ZNa7sea00E9tHMxc+RAPjs7iz+8s7/X3A/lZrJZC+SISLZzJbIEWN6yU1UrVDVNVbNUNQt4F1ioqoUEus6ukoBEAokoOMXfSKsuNBEZHLT5CWCr83g5sEREYkUkG8gB3uvON2p6n9W+Ui7JuvAukXAxf8Ig6pua+ecu60I+H/lFJV0qUXMuvvrhcWSnJfLVZZs5VdvQbc/rFdeSjao2AncAbwA7gBdUdZuIPCgiCzs5/TGgH4GEsRZ4UlU3A4hIAjAPeLnVOQ87U6U3A1cCX3Hi2Aa8AGwHXgduV9XeN6/QdJsTp2rZdfxUr6ka0BXTL0plQGIMK60r7Zw1NyurivzM6WKJmq5qGU87WlHDd/6yo9ue1ytRbj65M/34tVZtD7Rz7BVBj6sITH9u67hq4AMd6ap6SwdxfAf4TpeCNn1eSxn+cF1S4HxERghXj8vgjW3HaGhqJjrS7vfuqu1HKyk9Xd8t4zWtTb8olf83dyS//ucePjxxEFeOzej21+gp9htlTCsFPj/946KYMCTZ61B61LzcTCprG3vdlFu3tYzXXH4OJWrOxVeuGUNORj/ue2kzFdXh251mycaYIKpKga+Uy0YNJDKMloDuDnNy0omLjmDFNutKOxdnStT0P7cSNV0VFx3Jj2+Ygr+qnm/9eZsrr9ETLNkYE+RAWTWHy2vOqZBibxEfE8mcnHRWbj+Oqq130xU19U2s3XvSlS60YJOGJXP7FaN4ecPhsP0yYMnGmCAFvsB4TV+aHBBsfm4mRypq2XakbyxVfKHeL1Hj/r15d1yVQ+7g/nztlS1hWe3Bko0xQQqK/WT2j2VUeqLXoXji6vGZRAis2H6884MNq5wSNTOy3a+fFxMVwSOLJ1NR08ADf9ra+QkhxpKNMY5mZwno2aPCewnoCzEgMYa8rAFh21XT0/KL/Mzowfuxcof0586rcnh181H+svloj7xmd7FkY4xj57FTlJ2uZ1YfHK8JNj83k53HTnGwrNrrUELa8crA/Vhuj9e09sUrRnHxsGS+8X9bKDlV16OvfSEs2RjjWF3csspi37m/pi3zcwcB1pXWmZZVOXtivCZYVGQEP1o8mdP1TXz9lS1hM5nDko0xjlU+PyPTExmcHO91KJ4aMTCBcYOSrJpAJ1YVlZDWL6ZbS9R0VU5mEkvnjWHF9uP838bDPf7658OSjTFAfWMz7+0t6/VLCnTVvNxM3ttbxskwnPXUE5qblVU+P5ePTiPCo/ux/n3OSKZflMp//2kbxypqOz/BY5ZsjAE2HSqnur6pz3ehtZifO4hmhb/vPOF1KCFpx7FK/FX1Pd6FFiwyQnhk8WTqm5q57+XNId+dZsnGGAIlakQCpd1NYI2bwclxtsZNO94fr/H2Sjg7LZF7F4zjrV0lvFB4sPMTPGTJxhgCSwpMHJJMSkKM16GEBBFhXm4mb+/2U9tgRdJbW1XkZ2ymeyVqzsVnLsti5sgBPPTqDg6dDN0ZhJZsTJ9XXd/IhoMn+8SqnOdiXm4mNQ1NrCrqHYt3dZea+ibe21fm+VVNi4gI4YeLJqOq3PvSZpqbQ7M7zZKN6fPe21tGQ5Pa5IBWLs0eSFJclHWltfLevjLqG5uZMyZ0lo8fPiCBr310PAW+Up5+74DX4bTJko3p81YXlxITGcElWe6XHAknMVERXDk2gzd3nKApRL8te2FVUUmgRE2I/b7cNGMEc3LS+N5rOzhQGnrdaa4mGxFZICK7RMQnIvd1cNwiEVERyXO2o0XkKWflzR0icr/TPlZENgb9VIrIXc6+H4rIThHZLCKviEiK054lIjVB5/zKzfdswk+Bz8/UESnEx/SNJaDPxfwJmZSermf9gZNehxIy8ov8XJKVGnK/LyLCD/7lYiJFuGfZppDrTus02YjIf4hIfwn4nYisF5H5XTgvksDyztcCucCNIpLbxnFJwJ3AmqDmxUCsqk4CpgNfEJEsVd2lqlNUdYrTXg284pyzEpioqhcDu4H7g56vuOU8Vb2ts9hN33HydD3bj1a6tvBVuPvQmHSiI4WVVk0AgBOVtew8dsrTKc8dGZISz39dl8t7e8t4cvU+r8M5S1eubD6nqpXAfCAd+Czw/S6cNwPwqeoeVa0HngOub+O4h4CHgeC7khRIFJEoIB6oB1rXPL+aQBLZD6CqK1S10dn3LjCsCzGaPu6dPaWo0ufrobUnKS6aWaPSWLHtWMjfx9ETQmXKc0cWTx/G1eMyePj1nRSXVHkdzhldSTYtt8d+BHhSVTcFtXVkKBA88fuQ0/b+E4tMBYar6qutzl0GnAaOAgeAR1S19Vq1S4Bn23ntzwF/DdrOFpENIvJPEZnT1gkicquIFIpIYUlJSUfvy/Qiq3x++sVGMXlY31oC+lzMy81kX2k1vhOh8x+XV1b5/KT1i2H8oP5eh9IuEeF7n5xEXHQk97y4KWTG27qSbNaJyAoCyeYNp9uruQvntZWQzrxrEYkAHgWWtnHcDKAJGAJkA0tFZGTQuTHAQuDFD7yoyNeBRuBpp+koMEJVpwJ3A8+IyAd+U1T1CVXNU9W89PTQvEQ23W+1z8+l2QOIirS5Mu2Zl5sJWGHO5mYlv8jPbA9L1HRVRv84Hrx+AhsOlPPE23u8DgfoWrL5PHAfcImqVgPRBLrSOnMIGB60PQw4ErSdBEwE3hKRfcBMYLkzSeAm4HVVbVDVE0ABkBd07rXAelU967dfRD4DfAz4tDrX/Kpap6qlzuN1QDEwpgvxm17ucHkN+0qrrQutE5n945g8PKXPJ5udx07hr6oL2fGa1hZOHsKCCYN4dOVudh075XU4XUo2lwG7VLVcRG4GvgFUdOG8tUCOiGQ7VyJLgOUtO1W1QlXTVDVLVbMIjLMsVNVCAl1nVzmTEhIJJKKdQc99I6260ERkAXCv8xzVQe3pzmQFnKujHCA0Ur3xVIHPlhToqvm5mWw6WM7xytAv+OiW/KJA93ooj9cEExG+/YmJ9IuLYumLG2lo6kqHlHu6kmweB6pFZDLwVWA/8IfOTnIG6+8A3gB2AC+o6jYReVBEFnZy+mNAP2ArgaT1pKpuBhCRBGAe8HKrc35B4GppZaspznOBzSKyicBY0G1tjP+YPmi10/8+NrPnS8SHm/lOV1pfnpW2yhcoUZMZAiVquiqtXyzf+fhEth6u5Jf/KPY0lqguHNOoqioi1wM/VdXfOd1VnVLV14DXWrU90M6xVwQ9riIw/bmt46qBD3wVVdXR7Rz/EvBSV+I1fYeqUlBcymV9eAnoczE6ox/ZaYms2H6cm2de5HU4Pa62oYk1e8u4JQzf+7WTBrNw8hB+/vcirsnNYMIQbybDdOXK5pRzU+UtwF+cLqlod8Myxl2+E1WUnKpj9ijrQuuKlsKc7xT7OVXb4HU4Pe69vU6JmjDpQmvtWwsnkJoYw9IXNlHf6E13WleSzaeAOgL32xwjMH35h65GZYzL3h+vCc//PLwwPzeThiblrV1979aA/KISYiIjuDQ7PL+cpCbG8L1PTGLnsVP87M0iT2LoNNk4CeZpIFlEPgbUqmqnYzbGhLKC4lKGD4hn+IAEr0MJG1NHpDIwMaZPjtvkF/m5JDv0StSci2tyM/mXacN4/J/FbDpY3uOv35VyNTcA7xEYQ7kBWCMii9wOzBi3NDY18+6eUitRc44iI4Rrxmfyj50nPOuK8UJLiZrLR4fHlOeOPHBdLun9Yln64qYeX6eoK91oXydwj81nVPVfCdxw+V/uhmWMe7YcruBUbSOzbEmBczYvN5NTdY2s2VvqdSg9ZpUv9EvUdFVyfDQ/WHQxvhNV/Hjl7h597a4kmwjnxsoWpV08z5iQtLo48B/lLJsccM4uz0kjPjqSFdv6TldafpGfgYkx5A4O3RI15+JDY9K5ccYIfpO/h8J9PXcXSFeSxusi8oaI/JuI/BvwF1pNZzYmnBT4/IwblMTAfrFehxJ24qIjmTsmjZXbj/eJwpyq4VOi5lx8/aPjGZoSzz0vbqK6vrHzE7pBVyYI/CfwBHAxMBl4QlXvdTswY9xQ29BE4f6TNgvtAszPHcSxylq2HO5KIZHw9n6Jmt71+9IvNoqHF13MvtJqHn59V4+8Zldu6rQbI02vsW7/Seobm61EzQW4alwGkRGBNW4uHpbidTiuer9ETfhPDmht1qg0/m1WFv+zeh/zJ2S6PobZ7pWNiJxyVsJs/XNKRFqvLWNMWCjw+YmKEGaE6f0SoSA1MYZLslL7xLhNfpGfMZn9GJQcPiVqzsVXF4wla2ACX122mao6d7vT2k02qpqkqv3b+ElS1d4xUmb6nILiUiYPT6FfbJcu6k075uUOYtfxU+wvPe11KK6pbWjivb1lvWLKc3sSYqJ4ZPFkDpfX8N3Xdrj6WjarzPQZFTUNbDlUbiVqukFfKMy5dl8ZdY3NzBnTu8ZrWsvLGsC/X57NM2sO8PZu96pDWLIxfcaaPaU02xLQ3WL4gATGDUrq1Wvc5Bf5nRI1A7wOxXVL549lVHoi9760mYoad2rfWbIxfcbq4lLioiOYOqJ3D2r3lPkTBlG4r4yy0/Veh+KKt3eXkJeVSkJM7+9yjYuO5Ec3TOF4ZS0PvbrdldewZGP6jFU+PzOyBxIbFb71rULJ/NxMmhXe3NH7rm5OnHJK1PSyKc8dmTI8ha99ZDwfnTTYlefvSm20tmalHRSRV5yVLzs6d4GI7BIRn4jc18Fxi0REnSWhEZFoEXlKRLaIyA5niQNEZKyzMFrLT6WI3OXsGyAiK0WkyPkz1WkXEfmZE8NmEZl2Lh+Q6R2OV9biO1Fl4zXdaMKQ/gxJjuuVXWktVcHn9sIpzx359zkjuXJchivP3ZUrmx8D/0lgaYFhwD3Ab4DngN+3d5Kz7s1jwLVALnCjiOS2cVwScCewJqh5MRCrqqfmZKcAACAASURBVJOA6cAXRCRLVXep6hRVneK0VwOvOOfcB7ypqjnAm842zuvnOD+3Elh51PQxq4ttSYHu1rLGTX5RCTX1PVvU0W35u/0M6EUlakJBV5LNAlX9taqeUtVKVX0C+IiqPg+kdnDeDMCnqntUtZ5Acrq+jeMeAh4Gghc3VyBRRKKAeKAeaH1vz9VAsarud7avB55yHj8FfDyo/Q8a8C6QIiLuXCeakFXgKyUlIdr+8+hm8ycMorah+czNj72BqpLv83N5LytR47WuJJtmEblBRCKcnxuC9nVUHGkocDBo+5DTdoaITAWGq+qrrc5dBpwGjgIHgEdUtXXFuCXAs0Hbmap6FMD5s+VasNM4TO+mqqz2+bls5ED7z6ObzcgeQP+4qF41BXrX8VOUnKrrU+M1PaEryebTBJaEPuH83ALcLCLxwB0dnNfWv+ozyUlEIoBHgaVtHDcDaAKGANnA0uDxIRGJARYCL3Yh/g7jCHrOW0WkUEQKS0p6z7c0A/tKqzlSUWtTnl0QHRnBVeMyeHPnCZqae0dhzvzdvWdJgVDSlUKce1T1OlVNc36uU1Wfqtao6qoOTj0EDA/aHgYcCdpOAiYCb4nIPmAmsNyZJHAT8LqqNjjLGxQAeUHnXgusV9Xgr1PHW7rHnD9blkXoLI6W9/mEquapal56et8aFOztziwBbZMDXDEvdxBlp+tZt/+k16F0i7eLSsjJ6Mfg5HivQ+lVOp1ALiLDgJ8DswlcEawC/kNVD3Vy6logR0SygcMEur1uatmpqhXAma8OIvIWcI+qForI1cBVIvJHIIFAIvpJ0HPfyNldaADLgc8A33f+/FNQ+x0i8hxwKVDR0t3WGzU2NVNZ20h5dT0VNQ2U1zRQUd0QeFzdQHlN/fvbNQ3MHjWQr8wbg0jv7V5aXexncHIc2WmJXofSK31obDoxkRGs2HaMGWF+A2RLiZpPX3qR16H0Ol25W+lJ4BkCM8QAbnba5nV0kqo2isgdwBtAJPB7Vd0mIg8Chaq6vIPTH3NeYyuBbrAnVXUzgIgkOK/9hVbnfB94QUQ+T2CcpyXe14CPAD4Cs9c+2+k79piqUtvQ7CSE+kCSqG6gMni7JpAwKqrfb6uoaeBUbcfF9JJio0hOiCY5PhoR+NnffaT3j+OWmb3zH1dzs/JOcSlXjcvs1QnVS/1io5g1eiArdxzn6x8dH9afc+G+k4ESNdaF1u26kmzSVfXJoO3/abm3pTOq+hqtFlpT1QfaOfaKoMdVvJ8sWh9XDXygP0RVSwnMUGvdrsDtXYm3uzU3K6dqG89KBoErjbO3A4/rg64+Gjpc4z0yQkiJjyY5IZqU+GjS+8WSk5FEcnw0KU5bYF/MmWOS46PpHx9NdGTEWfF9/qm1PPjnbeQO7s/0izqaXBieth+t5GR1gy0p4LL5uYP42itb2H28irGDkrwO57zlF5UQHSlcOjK8r9BCUVeSjV9Ebub9bqsbCSwNbVrZeaySe5dtPpNAKmsb6Ggxw/joSFKcq4yUhGhGpvU78/hMsmjZbkkkCTEkxkR2y7fHiAjhJ5+aynW/WMWXnl7Hn798ORlJvauU+pnxGpsc4KprxmfwtVdg5fZjYZ1s3i7yk3fRgD5RoqandeUT/RzwCwIzxxRYTRh0RXkhPjqS5IQYLhqYeCZBBJJEzFlXIi3dWKFQNiU5IZpf3zKdT/yygDue2cDT/37pWVc/4a6guJTRGf3I7N+7kmioyegfx9QRKazYfpw7rsrxOpzzUnKqjh1HK/nqgrFeh9IrdZpsVPUAgWnGZzjdaD9p+4y+66KBifzhczO8DuOcjR/cn+9/8mLuen4j33ttJw9c94FCD2GpvrGZtXvLuCFvmNeh9AnzcjN5+PVdHK2oCcuZXC1XwXN68fo1Xjrfr7B3d2sUxnMfnzqUf5uVxe8L9vKnjYe9DqdbbDhwkpqGJru/pofMzx0EwN/C9AbPt4tKSE2IZsIQqzLhhvNNNuE73cS06+sfHc8lWanc99IWdh4L/5W/C4pLiRCYOdImB/SE0Rn9GJmWGJaFOVWVVUV+Ls9JtyoTLjnfZNM7bhU2Z4mOjOCxm6aRFBfFF/53nWuLKPWU1T4/k4Ymkxwf7XUofca8CZm8U1wadr87u49XceJUnU15dlG7yaadpQUqReQUgTIyphfK6B/H4zdP4/DJGu5+fiPNYVqC5HRdIxsPllsXWg+bn5tJY7Py1q4TnR8cQloKiVqycU+7yUZVk1S1fxs/Sapq8wJ7sekXDeCB63J5c+cJfvEPn9fhnJf39pbR2KzMHmX/efSkKcNTSesXG3aFOd8u8jPaStS4qvfMcTXd6paZF/HJqUN59G+7+UeYfUuFwMyimKgI8rJ6342qoSwyQrhmfAZv7SqhrjE81ripbWhizZ5Su6pxmSUb0yYR4TufmMS4Qf35j2c3cKC02uuQzklBcSnTR6QSF+39vUx9zfwJmVTVNfLuntargoSmdfutRE1PsGRj2hUfE8mvb56OiPCFP64Lm9UYS6sCN+dZiRpvzBqVRkJMJCu2HfM6lC55u6VETbb9vrjJko3p0IiBCfx0yRR2Hqvka69sQTuqvxMiVhcHqilZiRpvxEVH8qEx6fxtx/GwmGCSv9vP9ItSSYy1oWg3WbIxnbpibAZfuWYMr2w4zB/e2d/5CR5bXewnKTaKSUOTvQ6lz5qXm8nxyjo2H67wOpQOlZyqY/vRSubkWNUAt1myMV1yx5WjuWZ8Bg+9up3CfaHdF1/gK+XSkQOJ6kU13sLNVeMyiIwQVm4P7a601cW2KmdPsX+NpksiIoQf3TCFYanxfPHp9ZyorPU6pDYdLKvmQFm1jdd4LCUhhhlZA1ixLbSnQL+92++UqLGrYLdZsjFdlhwfza9vyaOqtpHbn1lPQ1P7a+54peWbqo3XeG/+hEyKTlSx13/a61DapKrkF5Uwe3QakVaixnWuJhsRWSAiu0TEJyL3dXDcIhFREclztqNF5CkR2SIiO0Tk/qBjU0RkmYjsdPZd5rQ/LyIbnZ99IrLRac8SkZqgfb9y8z33dmMHJfGDRRezdt9JvvOXHV6H8wEFvlLSk2LJyejndSh93rzcTICQ7UprKVEz18ZreoRr0y9EJJLA8s7zgEPAWhFZrqrbWx2XBNwJrAlqXgzEquokZxno7SLyrKruA34KvK6qi0QkBkgAUNVPBT3nj4DgkcliVZ3S7W+yj1o4eQibDpbzu1V7mTw8mU9MDY0S/qrK6uJSZo8eGNZLE/cWw1ITyB3cn5Xbj3Pr3FFeh/MBLSVqLrfxmh7h5pXNDMCnqntUtR54Dri+jeMeAh4GggcBFEgUkSggHqgHKkWkPzAX+B2Aqtarannwk0ngf5kbeH9lUeOC+64dx4zsAdz/8ha2HwmNCtG7j1fhr6qzEjUhZF5uJoX7T+KvqvM6lA/IL/IzKj2RISlWoqYnuJlshgIHg7YPOW1niMhUYLiqvtrq3GXAaeAocAB4RFXLgJFACfCkiGwQkd+KSGKrc+cAx1W1KKgt2zn+nyIyp61gReRWESkUkcKSkpJzfKt9T0uF6OT4aG774zoqqr2v8tuy+NUsmxwQMuZPyEQV/r4jtEoe1TY0sWZvqU157kFuJpu2+jHO3OElIhEElppe2sZxM4AmAtWls4GlIjKSQLffNOBxVZ1KICG1Hgu6kbOvao4CI5zj7waeca6Qzg5M9QlVzVPVvPR0+wXsivSkWB6/eTpHK2q46/kNnt/At7rYz0UDExiWmuBpHOZ9uYP7MzQlnhUhNm6zbv9JahuamTvGroJ7ipvJ5hAwPGh7GHAkaDsJmAi8JSL7gJnAcmeSwE0ExmUaVPUEUADkOc95SFVbxneWEUg+ADjdbp8Enm9pU9U6VS11Hq8DioEx3fg++7RpI1J54LoJ/GNXCT99s6jzE1zS2NTMu3vKbBZaiBER5uVmkl/kp7q+0etwzsgv8luJmh7mZrJZC+SISLYzkL8EWN6yU1UrVDVNVbNUNQt4F1ioqoUEus6ukoBEAolop6oeAw6KyFjnaa4GgiccXOMcd6ilQUTSnckKOFdHOcAel95zn3TzpSNYNH0YP32ziDd3eHNfxaZDFVTVNdp4TQian5tJXWMzb+/2ex3KGflFJUwbYSVqepJryUZVG4E7gDeAHcALqrpNRB4UkYWdnP4Y0A/YSiBpPamqm519XwaeFpHNwBTgu0HnLeGDEwPmAptFZBOBK6HbnPEf001EhG9/fCITh/bnruc3ss+D+ypWO+M1l42yb6qh5pLsASTHR4fMGjf+qjq2Halk7hjrLu9JrqZ1VX0NeK1V2wPtHHtF0OMqAtOf2zpuI4Eutbb2/VsbbS8BL3U1ZnN+4qIjefzT07nuF6u47Y/rePlLs0iI6blvjQXFfnIH92dAYkyPvabpmujICK4al8GbO4/T2NTseRmhlokkVqKmZ1kFAdNthg9I4GdLprLr+Cnuf7nnKkTX1Dexfn+5lagJYfNzMymvbqBw/0mvQyG/yE+KlajpcZZsTLeaOyade+aP5U8bj/Bkwb4eec3C/WXUNzUzyyYHhKy5Y9KJiYrwvFaalajxjiUb0+2++KFRzM/N5Luv7WDNnlLXX6/AV0pUhDAja4Drr2XOT2JsFJePTmPljmOerolUdKKK45V1zLUutB5nycZ0u0CF6MmMGJDA7c9s4LjLFaJXF/uZOiLFZhaFuHm5mRwsq2HnsVOexfD27pYSNTY5oKdZsjGuSIqL5te3TKe6vpEv/nEd9Y3uVIiuqG5gy+EKZtmU55B39fgMRPB0Vtoqn5+R6YkMtRI1Pc6SjXFNTmYSP1w0mfUHyvn2X7Z3fsJ5eGdPKaq2pEA4yEiKY9qIVM+qCdQ1NvHunlKr8uwRSzbGVR+9eDC3zh3JH97Zz0vrDnV+wjkq8PmJj45kyvCUbn9u0/3m5Way9XAlR8prevy11+0LlKixKc/esGRjXPfVD4/lspED+dorW9jazWvSFxT7uXTkAGKi7Fc5HMw/s8ZNz3el5fsCJWpmjrQp8l6wf6HGdVGREfz8pqkMSIzhtj+uo7y6vlue91hFLXtKTluJmjAyMr0fo9ITvUk2RSVMtRI1nrFkY3pEWr9AhegTlXXc+dxGmrqhQrQtKRCe5k8YxLt7Sqmo6bllKUqr6th6uNKmPHvIko3pMVOGp/DNhRN4e3cJP/nb7gt+voJiPwMSYxg/6AMrRpgQNi83k8Zm5a1dPbfGzaozJWpscoBXLNmYHnXjjOF8Km84P/+774K6UlSV1b5SLhs5kAi7EzysTBmWQnpSbI9WE1hV5Cc5PpqJQ61EjVcs2ZgeJSJ86/oJXDwsmbuf38iekqrzep49/tMcq6y1LrQwFBEhXDM+k7d2naCuscn11wuUqPFzuZWo8ZQlG9Pj4qIj+eWnpxEVKdz2x3Wcrjv3RbValhSwyQHhaf6ETE7XN7G62P1yRr4TVRyrrLUpzx6zZGM8MSw1gZ/fOA3fiSrufWnzOdfLKvCVMjQlnosG2hLQ4WjWqIEkxkT2yKy0t4sCX0wut2TjKUs2xjOX56Txnx8ex6ubj/K7VXu7fF5Ts/LOnlJmjRqIiHWLhKPYqEiuGJvByu3Hae6GmYkdWVVUwsi0RIal2hcTL7mabERkgYjsEhGfiNzXwXGLRERFJM/ZjhaRp0Rki4jsEJH7g45NEZFlIrLT2XeZ0/5NETksIhudn48EnXO/E8MuEfmwm+/ZnJvbPjSSBRMG8b2/7uSdLnapbD9SSUVNg5WoCXPzcjMpOVXHpkPlrr1GoERNmXWhhQDXko2IRBJY3vlaIBe4UURy2zguCbgTWBPUvBiIVdVJwHTgCyKS5ez7KfC6qo4DJhNYcrrFo6o6xfl5zXn+XALLRU8AFgC/dGIzIUBE+OHii8kamMCXn13P0YrOy5issvtreoUrx2YQFSGscLErbd3+k9Q0NNmU5xDg5pXNDMCnqntUtR54Dri+jeMeAh4GguvQK5AoIlFAPFAPVIpIf2Au8DsAVa1X1c6+Fl0PPKeqdaq6F/A5sZkQEagQnUdNfRNf/OP6TmcorS72MyazHxlJcT0UoXFDckI0l44c4Oq4TX6Rn6gIYeYo+2LiNTeTzVDgYND2IaftDBGZCgxX1VdbnbsMOA0cBQ4Aj6hqGTASKAGeFJENIvJbEUkMOu8OEdksIr8XkdSuxuHEcquIFIpIYUlJyTm/WXNhRmf045HFk9l4sJwH/9x+hei6xibW7iuzJQV6ifm5g/CdqDrvKfCdWVXkZ9qIVPpZiRrPuZls2hq5PTMSKCIRwKPA0jaOmwE0AUOAbGCpiIwEooBpwOOqOpVAQmoZC3ocGAVMIZCkftSVOM40qD6hqnmqmpeebpfcXrh20mBu+9Aonl5zgBcKD7Z5zPr95dQ2NNt4TS9xjYuFOUur6th6pMLGa0KEm8nmEDA8aHsYcCRoOwmYCLwlIvuAmcByZ5LATQTGZRpU9QRQAOQ5z3lIVVvGd5YRSD6o6nFVbVLVZuA3vN9V1lkcJoTcM38Ms0cP5Bv/t5Uthz5YIXp1sZ8IgUtH2hLQvcHQlHgmDu3vyrhNQXFgraM5Y+zLYyhwM9msBXJEJFtEYggM0i9v2amqFaqapqpZqpoFvAssVNVCAl1nV0lAIoFEtFNVjwEHRWSs8zRXA9sBRGRw0Gt/AtjqPF4OLBGRWBHJBnKA91x6z+YCRUVG8LMlU0nvF8ttf1zHydNnV4gu8Pm5eFgK/eOiPYrQdLd54wex/sBJSk7Vdevz5u8uITk+mklWoiYkuJZsVLURuAN4g8CMsRdUdZuIPCgiCzs5/TGgH4GEsRZ4UlU3O/u+DDwtIpsJdJl912l/2JkqvRm4EviKE8c24AUCSel14HZVdb9GhjlvA/vF8vjN0yipquPO5zacqRB9qraBTYcqmG2z0HqV+RMyUYU3d3Tf1Y2qssrnZ/bogVaiJkS4OmrmTD9+rVXbA+0ce0XQ4yoC05/bOm4jgS611u23dBDHd4DvdCloExIuHpbCQ9dP4N6XtvCjFbv46oJxvLe3jKZmtRI1vcy4QUkMS41nxfbjLJkxolues7ikiqMVtdxpU55Dhk3RMCHrU5eMYOPBCn75VjGTh6ewZk8ZsVERTLsotfOTTdgQEebnDuKPa/Zzuq6xWxY3e3u3U6LGJpKEDCtXY0LaNxfmMnl4Cktf2MQb246Rl5VKXLTdk9vbzMvNpL6xmbd3d89tB/lOiZrhA6xETaiwZGNCWmxUJI9/ehqxUREcLq+x+2t6qUuyUklJiO6WKdAtJWqs8GZosWRjQt6QlHh+ftNUhiTH8eEJg7wOx7ggKjKCq8Zl8ObOEzQ0NV/Qc63fX24lakKQJRsTFmaNSmP1/VczOqOf16EYl8zPHURFTQNr95Vd0PPkF5UEStTYvVghxZKNMSYkzB2TRmxUxAUvF53vlKhJsnuxQoolG2NMSEiIiWJOThortx8/58X0WpSdrmfrkQobrwlBlmyMMSFjXm4mh8tr2H608rzOL/D5AyVqLNmEHEs2xpiQcfX4TETOvzBnflEJ/eOiuHhYSjdHZi6UJRtjTMhI6xfL9BGp5zVuo6rkF/m5PCfNStSEIEs2xpiQMn9CJtuPVnLoZPU5nVdccpqjFbVcPtqmPIciSzbGmJAyLzdwL9W5dqXlFwWqD9h4TWiyZGOMCSnZaYnkZPQ7j2TjJ9tK1IQsSzbGmJAzLzeTNXvLKK+u7/xgoL6xmXf3lFrhzRBmycYYE3LmTxhEU7Pyj10nunT8+gMnqa5vsi60EGbJxhgTci4emkxGUmyXZ6XlF5UQGSFcNsoW1gtVriYbEVkgIrtExCci93Vw3CIRURHJc7ajReQpZ+XNHSJyf9CxKSKyTER2Ovsuc9p/6LRtFpFXRCTFac8SkRoR2ej8/MrN92yMuXAREcK83Ez+ubuE2obOF9YNlKhJsRI1Icy1ZCMikQSWd74WyAVuFJHcNo5LAu4E1gQ1LwZiVXUSMB34gohkOft+CryuquOAyQSWnAZYCUxU1YuB3cD9Qc9XrKpTnJ/buuktGmNcNC83k+r6JlYX+zs87uTperYcrrApzyHOzSubGYBPVfeoaj3wHHB9G8c9BDwM1Aa1KZAoIlFAPFAPVIpIf2Au8DsAVa1X1XLn8QpVbXTOfxcY5sJ7Msb0kMtGDaRfbFSns9IKip0SNWNsvCaUuZlshgIHg7YPOW1niMhUYLiqvtrq3GXAaeAocAB4RFXLgJFACfCkiGwQkd+KSGIbr/054K9B29nO8f8UkTltBSsit4pIoYgUlpR0z2qBxpjzFxsVyYfGprNy+wmam9svzJm/2x8oUTM0uQejM+fKzWTTVr2IM78xIhIBPAosbeO4GUATMATIBpaKyEggCpgGPK6qUwkkpLPGgkTk60Aj8LTTdBQY4Rx/N/CMc4V0dmCqT6hqnqrmpafb5bgxoWB+bib+qjo2HCxvc3+gRE0Js0enERVp851CmZt/O4eA4UHbw4AjQdtJwETgLRHZB8wEljuTBG4iMC7ToKongAIgz3nOQ6raMr6zjEDyAUBEPgN8DPi0OjXKVbVOVUudx+uAYmBMN79XY4wLrhibQVSEsGL7sTb37/Gf5khFrS0pEAbcTDZrgRwRyRaRGGAJsLxlp6pWqGqaqmapahaBcZaFqlpIoOvsKglIJJCIdqrqMeCgiIx1nuZqYDsEZr4B9zrPcaaokoikO5MVcK6OcoA9Lr5vY0w3SY6P5rJRA9sdt8nfHejynmtLQIc815KNM1h/B/AGgRljL6jqNhF5UEQWdnL6Y0A/YCuBpPWkqm529n0ZeFpENgNTgO867b8gcLW0stUU57nAZhHZROBK6DZn/McYEwbm5Wayp+Q0vhNVH9iXX+Qna2CClagJA1FuPrmqvga81qrtgXaOvSLocRWB6c9tHbeRQJda6/bR7Rz/EvBSl4M2xoSUa8Zn8sCftrFy+3FGZ/Q7017f2Mw7e0r5l2k28TQc2IiaMSakDUmJZ9LQ5A+M22xwStTYeE14sGRjjAl583Mz2XiwnBOV79+Ol1/ktxI1YcSSjTEm5M2bkIkq/G3H+4U584tKmDo8hf5WoiYsWLIxxoS8sZlJjBiQwEqnK+3k6Xo2H65gjs1CCxuWbIwxIU8kUJizwFdKVV0jq4tLUcXGa8KIJRtjTFiYn5tJfVMzb+8uIb+ohKS4KCYPsxI14cLVqc/GGNNdpl+USmpCNCu2HWPtvpPMHmUlasKJ/U0ZY8JCVGQEV4/P5LWtxzhcXmNVnsOMJRtjTNiYl5tJfWMzAHNs/ZqwYt1oxpiwMTcnnbjoCDL7xzFioJWoCSeWbIwxYSM+JpKl88aSmhjjdSjmHFmyMcaElf83d6TXIZjzYGM2xhhjXGfJxhhjjOss2RhjjHGdq8lGRBaIyC4R8YnIfR0ct0hE1FkSGhGJFpGnRGSLiOwQkfuDjk0RkWUistPZd5nTPkBEVopIkfNnqtMuIvIzJ4bNIjKt7SiMMca4xbVk4yzF/BhwLZAL3CgiuW0clwTcCawJal4MxKrqJGA68AURyXL2/RR4XVXHAZMJrAIKcB/wpqrmAG862zivn+P83Ao83k1v0RhjTBe5eWUzA/Cp6h5VrQeeA65v47iHgIeB2qA2BRJFJAqIB+qBShHpT2CZ598BqGq9qpY751wPPOU8fgr4eFD7HzTgXSBFRAZ315s0xhjTOTeTzVDgYND2IaftDBGZCgxX1VdbnbsMOA0cBQ4Aj6hqGTASKAGeFJENIvJbEUl0zslU1aMAzp8ZXY3DieVWESkUkcKSkpJzf7fGGGPa5WaykTba9MxOkQjgUWBpG8fNAJqAIUA2sFRERhK4L2ga8LiqTiWQkNodC+pKHGcaVJ9Q1TxVzUtPtzIYxhjTndy8qfMQMDxoexhwJGg7CZgIvCUiAIOA5SKyELiJwLhMA3BCRAqAPOBt4JCqtozvLOP9ZHNcRAar6lGnm6xlSb/O4viAdevW+UVk/zm927OlAf4LOL8nhVOsEF7xWqzuCad4wylWuLB4L2pvh5vJZi2QIyLZwGFgCYEkAoCqVhB4UwCIyFvAPapaKCJXA1eJyB+BBGAm8BNVPSYiB0VkrKruAq4GtjtPsRz4DPB9588/BbXfISLPAZcCFS3dbe1R1Qu6tBGRQlXNu5Dn6CnhFCuEV7wWq3vCKd5wihXci9e1ZKOqjSJyB/AGEAn8XlW3iciDQKGqLu/g9MeAJ4GtBLrBnlTVzc6+LwNPi0gMsAf4rNP+feAFEfk8gXGexU77a8BHAB9QHXS8McaYHuJqbTRVfY3Af/bBbQ+0c+wVQY+reD9ZtD5uI4EutdbtpQSudFq3K3D7ucRtjDGme1kFAXc84XUA5yCcYoXwitdidU84xRtOsYJL8Urgi78xxhjjHruyMcYY4zpLNsYYY1xnyeYCicg+p2DoRhEpdNraLArqtXZi/aaIHHbaNorIR7yOs0VbRVdD+LNtK9aQ/GxFZGxQTBtFpFJE7grFz7aDWEPyswUQka+IyDYR2Soiz4pInIhki8ga57N93plN67l2Yv0fEdkb9NlO6ZbXsjGbCyMi+4A8VfUHtT0MlKnq9yVQ7TpVVe/1KsaguPbxwVi/CVSp6iNexdUeEXkKyFfV3zr/OBOArxGan21bsd5FiH62LSRQMPcwgXvQbicEP9sWrWL9LCH42YrIUGAVkKuqNSLyAu/ffvGyqj4nIr8CNqmqp0WBO4j1CuBVVV3Wna9nVzbuaK8oqOkiab/oash9th3EGg6uBopVdT8h+Nm2EhxrKIsC4iVQSDiBQI3HqwhUPIHQ+mxbx9phdZULYcnmwimwQkTWicitTlt7RUG91lasEKiwsFlEfh8KXSeO9oquhuJn21GB2FD8bIMtAZ51HofiZxssOFYIwc9WVQ8DjxC4sfwooew1pQAABL5JREFUUAGsA8pVtdE5rM1iwD2trVhVdYWz+zvOZ/uoiMR2x+tZsrlws1V1GoF1c24XkbleB9SBtmJ9HBgFTCHwC/cjD+MLdj5FV73SXqyh+tkC4HT3LQRe9DqWzrQRa0h+tk7Su55AAeEhQCKBf2+teT5+0VasInIzcD8wDrgEGAB0S1eqJZsLpKpHnD9PAK8QqFh9XJw1c+TsoqCeaitWVT2uqk2q2gz8hkD8oeAQHyy6Oo3Q/GzbjDWEP9sW1wLrVfW4sx2Kn22Ls2IN4c/2GmCvqpY4hYRfBmYRWEerpWJLp8WAe0ibsarqUWf9rzoCZcO65bO1ZHMBRCRRAiuN4nSbzCdQz62lKCicXRTUM+3FKmcvJPcJAvF7TlWPAQdFZKzT1FJ0NeQ+2/ZiDdXPNsiNnN0tFXKfbZCzYg3hz/YAMFNEEkREeP/39h/AIueYUPls24p1R9AXDiEwttQtn63NRvv/7d1PiJVVGMfx788UVBQjx6UiiSQoGowi6sKlCymCFEFFBIsMchNtEwQlQjcqulBJIZAsUsGFjYKm0R+LcC4q1UrBhZsC/0CLUJ8W5xkc1Jn34p3jXPH32cy973vuec9c7szDPffyezqg0mPnRN4dCxyNiB2SpgJfAzPIUNBs/jZqhlnrl5StiABuAB80pWI/L/mVy0PA4NDVMXTZcwtDrnUP3fvcTqQ0FXw9E9jpxtctDLnWbn7dbgPWAPeBy8B7lM9ovqJsS10G1uc7h1E1xFpPA9MoIcj9wObMq+zsWi42ZmZWm7fRzMysOhcbMzOrzsXGzMyqc7ExM7PqXGzMzKw6FxszM6vOxcbsBZHR76uaR5p1HxcbMzOrzsXGrAOSZqo0SzuYTajOSJog6XtJC3NMT/YSQtJGSSclncoGVR9J+jjTon+R9Fqb1+2VdCETvPsGRYy8L+k3SS1J32YUyRSVxnljcsxESTcljZM0S9J3Oc8PkubkmNUqDbVaki5WefLspeJiY9a52cC+iJgL3AbebRg/D1hLCTjcAfybadE/AxuaLiZpHLAXWBURvcAXOQ+UBl2LImIB8AewKSNeWsDyHPMW0JfhiweALTnPJ8D+HLMVWJHzvN20JrMmY5uHmFmD6xHRn7d/B2Y2jD8fEfeAe5LuAKfy+BVgfhvXe4NSsM6WrEReocTsA8yTtB14FZgE9OXxY5QMrPOUvjD7JU2iJBJ/k/MADPQu+RE4kt0bj7exJrNhudiYdW5woOIDYAIl2HBg52D8MOMfDrr/kPb+JgVci4glTzl3BHgnIlqSNlJa/EJJdP4st+l6gXOUXiu3I+KJHvMRsVnSYmAl0C/pzYj4p421mT2Vt9HM6rhB+acOj6LlR8pfwDRJS6Bsq0mam+cmA7dyq23dwAMytfdXYDelv/yDiLgLXJe0OueRpAV5e1ZEXIqIrcDfwPQR/h3sJeNiY1bHLuBDST8BPSM5cUT8Rylgn0tqUWLgl+bpT4FLwFngz8ceegxYnz8HrAM25TzXKJ0bAXZKuiLpKnCR8pmP2TNziwEzM6vO72zMzKw6f0HArMtI2gcse+zw7og4PBrrMRsJ3kYzM7PqvI1mZmbVudiYmVl1LjZmZladi42ZmVX3P0bngRfcEtV8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "# train_means = grid_search.cv_results_['mean_train_score']\n",
    "# train_stds = grid_search.cv_results_['std_train_score']\n",
    "n_leafs = len(num_leaves_s)\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis,-test_means)\n",
    "plt.xlabel('num_leaves')\n",
    "plt.ylabel('Log loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed: 17.7min remaining:  8.8min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48561976505170185\n",
      "{'min_child_samples': 40}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed: 25.1min finished\n"
     ]
    }
   ],
   "source": [
    "# 3.min_child_samples\n",
    "# 叶子节点数目： 75，共9类，平均每类8个叶子节点\n",
    "# 最少类别的样本数目大概200左右，用三折交叉验证乘以2/3再乘以subsamples采样,100个左右样本\n",
    "# 每棵树的样本数目数目最少的类的样本数目 200 *2/3 *0.7 = 100\n",
    "# 有9类\n",
    "# 所以每个叶子节点约100/8=12个样本点\n",
    "# 搜索范围10-50\n",
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.1,\n",
    "         'max_bin':127,\n",
    "         'subsample':0.7,\n",
    "         'bagging_freq':1,\n",
    "         'colsample_bytree':0.7,\n",
    "         'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75}\n",
    "min_child_samples_s = range(10,50,10)\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "tuned_parameters = dict(min_child_samples=min_child_samples_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='neg_log_loss',verbose=5,refit=False)\n",
    "grid_search.fit(x_train,y_train)\n",
    "\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV9Z3/8dcnK2RlSdiSYFgCJKIIXinValGxolCsVajaTjszbW1rnVqXturY/n4/7TI6VtuZsZ3axTozHRWXttZ9waVulCCiQNi3hAQIW8IWsn1+f9wLBgwQsp27vJ+PBw8555577+f7OHje937P555j7o6IiCSepKALEBGRYCgAREQSlAJARCRBKQBERBKUAkBEJEGlBF3AicjLy/Pi4uKgyxARiRkLFy7c5u757T0WUwFQXFxMeXl50GWIiMQMM9twtMc0BSQikqAUACIiCUoBICKSoBQAIiIJSgEgIpKgFAAiIglKASAikqDiPgAam1u5//U1lK/fEXQpIiJRJe4DoLm1ld+9sZ7/95dltLbq3gciIgfFfQBkpKXwvYvG8sGmOp5YtCnockREokbcBwDAJRMKmFDUj7ueW87eA81BlyMiEhUSIgCSkowfzCxj6+4D/Oq1NUGXIyISFRIiAABOP6k/syYM41evr2XTrv1BlyMiEriECQCA7100DoA7n10ecCUiIsFLqAAo6NeXr50zkicXV7Nwg9pCRSSxJVQAAHztk6MYlJ3O7U9VqC1URBJawgVAZnoK350+jsWVu/jzYrWFikjiSrgAAPjsxAJOKcjlzmdXsK9RbaEikpgSMgCSkowffLqMzfUN3P/62qDLEREJRIcCwMymm9kKM1ttZjcfY7vLzczNLBRZTjOzB8zsAzNbbGZT22x7emT9ajP7NzOzLo/mBJxRPIAZpw7lP19bQ02d2kJFJPEcNwDMLBm4D7gIKAOuNLOydrbLBr4FzG+z+qsA7n4KcAHwUzM7+J6/BK4GSiJ/pnd+GJ1z8/RxtDrc9dyK3n5rEZHAdeQbwGRgtbuvdfdG4GHgkna2uwO4C2hos64MeBnA3bcCu4CQmQ0Fctz9bXd34L+Az3R+GJ1TNCCDr3xiBH9ctIlFG3f29tuLiASqIwFQAFS2Wa6KrDvEzCYCRe7+1BHPXQxcYmYpZjYCOB0oijy/6liv2ea1rzazcjMrr62t7UC5J+aac0eTl5XOHU8tI5xFIiKJoSMB0N7c/KEjZWRK517gxna2+x3hg3s58DPgLaD5eK952Er3+9095O6h/Pz8DpR7YrLSU/juhWN5d+Mu/vJ+Tbe/vohItOpIAFQR/tR+UCFQ3WY5GxgPvGpm64EpwJNmFnL3Zne/3t1Pc/dLgH7AqshrFh7jNXvVZacXcvKwHP7lmQoamlqCKkNEpFd1JAAWACVmNsLM0oArgCcPPujude6e5+7F7l4MvAPMcvdyM8sws0wAM7sAaHb3Ze5eA+w2symR7p8vAn/u5rF1WHKS8f2ZZVTXNfBrtYWKSII4bgC4ezNwLfA8UAHMdfelZna7mc06ztMHAe+aWQXwPeDv2jz2DeA3wGpgDfBsJ+rvNlNGDuSi8UP4xatr2FLfcPwniIjEOIulE5+hUMjLy8t77PU3bt/HtHte49MThvHTORN67H1ERHqLmS1091B7jyXkL4GPZvjADP7hE8U8/m4V71ftCrocEZEepQA4wrXnjiYvK01toSIS9xQAR8juk8qNnxrLgvU7eeaDzUGXIyLSYxQA7ZgTKmLckGx+rLZQEYljCoB2JEeuFrpp135++8a6oMsREekRCoCjOHNUHp8qG8wvXlnNVrWFikgcUgAcw60Xl9LY0srdL+hqoSISfxQAx1Ccl8nfn1nMowurWLKpLuhyRES6lQLgOK49r4T+GWoLFZH4owA4jty+qdxwwRjmr9vB80vVFioi8UMB0AFXnFHE2MHZ/OiZCg40qy1UROKDAqADUpKTuG1mKZU79vPAm+uDLkdEpFsoADro7JJ8zh83iP+Yt5ra3QeCLkdEpMsUACfg1hmlNDS1cM+LK4MuRUSkyxQAJ2BUfhZf/HgxjyzYyLLq+qDLERHpEgXACbru/BJy+qaqLVREYp4C4ATlZoTbQt9eu50Xl20JuhwRkU5TAHTCVZOHUzIoS22hIhLTFACdkJKcxD/PKGXD9n3811sbgi5HRKRTFACdNHXsIKaOzeff5q1i+x61hYpI7FEAdMFtM0rZ19jCvS+pLVREYo8CoAtGD8rm76acxP/O38iKzbuDLkdE5IQoALrouvNLyO6jtlARiT0KgC7qn5nGt6eV8MbqbcxbvjXockREOkwB0A2+MOUkRuZn8qOnK2hsbg26HBGRDlEAdIPU5CRum1HK2m17+Z931BYqIrFBAdBNzh07iLNL8vjZSyvZubcx6HJERI5LAdBNzIzvzyxjz4Fmfqa2UBGJAQqAbjRmcDaf/9hJ/M/8jazaorZQEYluCoBudv0FY8hIS+aHT1cEXYqIyDEpALrZgMw0rju/hNdW1vLKCrWFikj0UgD0gC9+vJgReeG20KYWtYWKSHRSAPSAtJQkbr24lNVb9/C/8zcGXY6ISLs6FABmNt3MVpjZajO7+RjbXW5mbmahyHKqmT1oZh+YWYWZ3dJm2+vMbImZLTWzb3d9KNFlWukgzho9kHtfWsmufWoLFZHoc9wAMLNk4D7gIqAMuNLMytrZLhv4FjC/zerZQLq7nwKcDnzNzIrNbDzwVWAyMAGYaWYlXR1MNDEzbptRRv3+Jn7+8qqgyxER+YiOfAOYDKx297Xu3gg8DFzSznZ3AHcBDW3WOZBpZilAX6ARqAdKgXfcfZ+7NwOvAZd2fhjRqXRoDldMHs5/v72B1Vv3BF2OiMhhOhIABUBlm+WqyLpDzGwiUOTuTx3x3MeAvUANsBG42913AEuAc8xsoJllABcDRe29uZldbWblZlZeW1vbkTFFlRsuGEPf1GR+/IzaQkUkunQkAKyddYeue2xmScC9wI3tbDcZaAGGASOAG81spLtXAHcCLwLPAYuB5vbe3N3vd/eQu4fy8/M7UG50yctK59rzRjNv+VZeXxl7ASYi8asjAVDF4Z/OC4HqNsvZwHjgVTNbD0wBnoycCL4KeM7dm9x9K/AmEAJw99+6+yR3PwfYAcTtRPnfn1XM8AEZ/PDpZTSrLVREokRHAmABUGJmI8wsDbgCePLgg+5e5+557l7s7sXAO8Asdy8nPO1znoVlEg6H5QBmNijy3+HAZ4GHunFcUSU9JZlbLy5l5ZY9PLSg8vhPEBHpBccNgMhJ2muB54EKYK67LzWz281s1nGefh+QRXjOfwHwgLu/H3nscTNbBvwF+Ka77+zsIGLBhScPZsrIAdzzwgrq9jcFXY6ICBZLtzEMhUJeXl4edBmdtrS6jpn//gZfPmsEt838SCetiEi3M7OF7h5q7zH9ErgXnTwsl8+Finjw7fWs27Y36HJEJMEpAHrZDZ8aQ1pyktpCRSRwCoBeNii7D988bzQvLtvCm6u3BV2OiCQwBUAA/vGsERT278sdTy2jpTV2zsGISHxRAASgT2q4LXT55t08orZQEQmIAiAgF40fwuTiAfz0hRXUN6gtVER6nwIgIAdvIr9jXyP3zVsddDkikoAUAAE6pTCXyycV8sCb69mwXW2hItK7FAAB+86FY0lJNn7yzPKgSxGRBKMACNignD5cM3UUzy3dzNtrtgddjogkEAVAFPjK2SMp6Ke2UBHpXQqAKNAnNZmbLxrHspp6HluotlAR6R0KgCgx89ShnH5Sf/71+ZXsOdDuvXFERLqVAiBKmBk/mFnGtj0H+MUragsVkZ6nAIgiE4r68dmJBfzmjXVU7tgXdDkiEucUAFHmO9PHkmzGvzyrtlAR6VkKgCgzNLcvX//kKJ7+oIa/rdsRdDkiEscUAFHo6nNGMjS3D7c/tZRWtYWKSA9RAEShvmnhttAlm+p5/N2qoMsRkTilAIhSsyYMY+Lwfvzr8yvYq7ZQEekBCoAodfBqoVt3H+A/X1sTdDkiEocUAFFs0vD+XHLaMO5/fS1VO9UWKiLdSwEQ5b43fRxmcOdzK4IuRUTijAIgyg3r15erzxnFXxZXs3CD2kJFpPsoAGLA1z85ksE56dz+l2VqCxWRbqMAiAEZaSl8b/o4FlfV8efFm4IuR0TihAIgRnzmtAImFOZy57Mr2NeotlAR6ToFQIxISgq3hW6ub+BXr60NuhwRiQMKgBgSKh7AzFOH8qvX11C9a3/Q5YhIjFMAxJibLxpHq8Ndz+lqoSLSNQqAGFPYP4Orzx7Jn96rZtHGnUGXIyIxTAEQg74xdRT52enc/tQy3NUWKiKdowCIQZnpKXz3wrEs2riLJxdXB12OiMSoDgWAmU03sxVmttrMbj7GdpebmZtZKLKcamYPmtkHZlZhZre02fZ6M1tqZkvM7CEz69P14SSOyyYVMr4ghzufXc7+xpagyxGRGHTcADCzZOA+4CKgDLjSzMra2S4b+BYwv83q2UC6u58CnA58zcyKzawgsm3I3ccDycAVXR1MIklKMr4/o4zqugZ+/Ve1hYrIievIN4DJwGp3X+vujcDDwCXtbHcHcBfQ0GadA5lmlgL0BRqB+shjKUDfyGMZgOYyTtDHRg7k4lOG8MtX17C5ruH4TxARaaMjAVAAVLZZroqsO8TMJgJF7v7UEc99DNgL1AAbgbvdfYe7bwLujqyrAerc/YX23tzMrjazcjMrr62t7ciYEsotF5XS0urc9bzaQkXkxHQkAKyddYdaT8wsCbgXuLGd7SYDLcAwYARwo5mNNLP+hL9FjIg8lmlmX2jvzd39fncPuXsoPz+/A+UmlqIBGXz57BE88e4m3q/aFXQ5IhJDOhIAVUBRm+VCDp+uyQbGA6+a2XpgCvBk5ETwVcBz7t7k7luBN4EQMA1Y5+617t4EPAGc2dXBJKprpo4iLyt8tVC1hYpIR3UkABYAJWY2wszSCJ+sffLgg+5e5+557l7s7sXAO8Asdy8nPMVznoVlEg6H5ZH1U8wsw8wMOB+o6NaRJZDsPqnc9KkxlG/YydMf1ARdjojEiOMGgLs3A9cCzxM+SM9196VmdruZzTrO0+8DsoAlhIPkAXd/393nEz4/8C7wQaSO+zs/DJkdKqJ0aA4/eWY5DU1qCxWR47NYmjIIhUJeXl4edBlR660127jq1/P5zoVj+ea5o4MuR0SigJktdPdQe4/pl8Bx5MxReVx48mDue2U1W+vVFioix6YAiDO3XlxKU0srd7+gm8iLyLEpAOLMSQMz+cezRvDowiqWbKoLuhwRiWIKgDj0zfNGMyAjTVcLFZFjUgDEoZw+qdzwqTH8bd0OnluyOehyRCRKKQDi1OdCRYwbks2Pn61QW6iItEsBEKdSkpP4/swyKnfs54E31wddjohEIQVAHDtrdB7TSsNtobW7DwRdjohEGQVAnPvnGaUcaG7hnhfVFioih1MAxLkReZl86ePFPLygkqXVagsVkQ8pABLAP51fQr++qdyhtlARaUMBkABy+6ZywwVjeGftDl5YtiXockQkSigAEsSVk4dTMiiLHz9TwYFmtYWKiAIgYRxsC92wfR8PvrU+6HJEJAooABLIOWPyOW/cIP795dVs36O2UJFEpwBIMLdeXMr+phbueXFl0KWISMAUAAlm9KAsvjDlJB7620aWb64PuhwRCZACIAF9e1oJ2X3UFiqS6BQACahfRhrXTyvhzdXbeblia9DliEhAFAAJ6vNTTmJUfiY/eqaCxubWoMsRkQAoABJUanISt80sY922vfz3OxuCLkdEAqAASGDnjh3EJ8fk8/OXVrJjb2PQ5YhIL1MAJLjbZpSyt7GFn72ktlCRRKMASHAlg7P5/MeG84f5G1m5ZXfQ5YhIL1IACN+eNobMtGS1hYokGAWAMCAzjeumjeGvq7bx6oraoMsRkV6iABAA/m7KSYzMy+SHTy+jqUVtoSKJQAEgAKSlJPHPM0pZU7uXP6gtVCQhKADkkPPGDeLskjzufWkVu/apLVQk3ikA5BAz47YZZexuaOJnL60KuhwR6WEKADnM2CHZXDl5OP/9zgZWb1VbqEg8UwDIR9xwwRgyUpP50dMVQZciIj1IASAfMTArnW+dX8IrK2p5baXaQkXiVYcCwMymm9kKM1ttZjcfY7vLzczNLBRZTjWzB83sAzOrMLNbIuvHmtl7bf7Um9m3u2dI0h2+dGYxxQMz+OFTy2hWW6hIXDpuAJhZMnAfcBFQBlxpZmXtbJcNfAuY32b1bCDd3U8BTge+ZmbF7r7C3U9z99Mi6/cBf+zyaKTbpKUkcevFpazauoeH/rYx6HJEpAd05BvAZGC1u69190bgYeCSdra7A7gLaGizzoFMM0sB+gKNwJH3ITwfWOPuaj6PMheUDebjIwdyz4srqdvXFHQ5ItLNOhIABUBlm+WqyLpDzGwiUOTuTx3x3MeAvUANsBG42913HLHNFcBDR3tzM7vazMrNrLy2VvPRvcnM+P7MMnbtb+Lf5qktVCTedCQArJ11h64YZmZJwL3Aje1sNxloAYYBI4AbzWxkm+emAbOAR4/25u5+v7uH3D2Un5/fgXKlO5UNy+GKM4p48K31rK3dE3Q5ItKNOhIAVUBRm+VCoLrNcjYwHnjVzNYDU4AnIyeCrwKec/cmd98KvAmE2jz3IuBdd9/S+SFIT7vhgrH0SU3mx88sD7oUEelGHQmABUCJmY2IfGK/Anjy4IPuXufuee5e7O7FwDvALHcvJzztc56FZRIOh7ZHkSs5xvSPRIf87HSuPW80L1Vs4Y1V24IuR0S6yXEDwN2bgWuB54EKYK67LzWz281s1nGefh+QBSwhHCQPuPv7AGaWAVwAPNGF+qWX/MNZxQwfkMEdagsViRsWSzcACYVCXl5eHnQZCevZD2r4xh/e5UeXjufzHzsp6HJEpAPMbKG7h9p7TL8Elg6bPn4Ik0cM4KcvrKS+QW2hIrFOASAdZmb8YGYZO/c18h/zVgddjoh0kQJATsj4glxmn17IA2+uY/22vUGXIyJdoACQE3bTp8aSlpzET57V1UJFYpkCQE7YoJw+XHPuaJ5fuoW31qgtVCRWKQCkU778iREU9OvLHU9V0NIaO51kIvIhBYB0Sp/UZG65eBwVNfU8Wl55/CeISNRRAEinzThlKKGT+nP3CyvYrbZQkZijAJBOMzN+8Okytu1p5Bevrgm6HBE5QQoA6ZJTC/tx2aRCfvvXdVTu2Bd0OSJyAhQA0mXfnT6W5CRTW6hIjFEASJcNzunDNVNH8cwHm5m/dnvQ5YhIBykApFt89ZyRDMvtw+1PLVNbaIzZuruBV1Zs5b5XVnP9I+/xwJvr2Lm3MeiypBekBF2AxIc+qcl876JxXPfwezz+bhVzQkXHf5L0qtZWZ8OOfSytrmNZdT1Lq+tZVlNP7e4Dh7bJy0rjj4s28ZNnlnNB2WBmhwo5uySf5KT2bgwosU4BIN1m1oRh/P6t9fzr8yuYccpQMtP1zysoB5pbWLVlT+RAX8fS6noqaurZ29gCQEqSMXpQFmeX5HHysFxOHpZD6dAccvumRn7bUcUfF1Xx9Ac1DMnpw+WnF3L56YUU52UGPDLpTrofgHSrRRt3cukv3uLac0dz04Vjgy4nIdQ3NLGsuv6wT/WrtuymOTIVl5mWTOnQHMqG5XDysBzKhuZSMjiLPqnJx3zdxuZWXq7YwtzySl5bWUurw8dGDGBOqIiLThlCRpoCPhYc634ACgDpdtc/8h5Pf1DDvBs/SWH/jKDLiRvuzpb6AyyrqWPppg8P9hvbtN/mZaWHD/KHDvY5FA/MJKmLUzib6xp4/N0qHi2vZP32fWSlp/DpCUOZHSpiYlE/zDRFFK0UANKraur2c+7drzKtdDD/cdWkoMuJSS2tzrpte1lWU39ozn5ZdT3b25ycLR6YETnQ54b/OzSHQTl9erQud2fB+p3MLa/k6fdr2N/UwuhBWcwJFXLpxELys9N79P3lxCkApNfd++JKfv7yKh77+scJFQ8Iupyo1tDUworNuw872FfU7GZ/U3i+PjXZGDM4m7KhkU/1w3IpHZpNdp/UQOvec6CZp9+vZm55FQs37CQlyTh33CDmhIqYOjaf1GQ1GUYDBYD0un2NzZx392sMyknnT9ec1eUpiHhRt6+JpTVtunCq61ldu+dQ62xWegplbefrh+VQMiibtJToPpiu3rqHRxdW8vjCTWzbc4C8rHQum1TA7FARowdlBV1eQlMASCD+uKiK6x9ZzD1zJvDZSYVBl9Or3J3quobDunCWVdezadf+Q9sMzkmPfKrPPXSwL+qfEdNh2dTSymsraplbXsm85VtpbnUmDe/HnFARM04dGvi3lkSkAJBAtLY6l/7yLTbX7eeVm6bGbddIc0sr67btZWnkYB+eyqln177wFVLNYERe5kcO9nlZ8T1fXrv7AH9atIlHyitZvXUPfVOTufiUocwJFTJ5xACdOO4lCgAJzMINO7nsl2/xrfNLuOGCMUGX02X7G1tYvrk+crAPd+Esr6nnQHMrAGkpSYwdnM3JbaZwxg3JSejfRLg771XuYm55FX9ZXM2eA80UD8xgdqiIyyYVMiS3Z09cJzoFgATqWw8t4vmlm5l301QK+vUNupwO27G38dAUzsFP9Wtr93DwShc5fVIOdeEcPNiPys/Syc9j2NfYzHNLNvPIgkrmr9tBksE5Y/KZEyri/NJBpKcc+7cJcuIUABKoTbv2c97drzJ9/BB+fsXEoMv5CHenauf+yDz9hwf7mrqGQ9sMy+1DWaQD52A3TmH/vprG6IL12/by2MIqHn+3ipq6BvpnpHLpxELmnFHIuCE5QZcXNxQAErifvrCCf5+3mieuOZNJw/sHVkdTSytravewdFP9YW2X9Q3NACQZjMrPOuxXs2XDchiQmRZYzfGupdX566paHi2v4oVlm2lqcU4tzGV2qIhZE4aR21cnjrtCASCB23ugmXPvfpVh/fryx2vO7JVPznsPNB+arz/Ydrliy24aI/P16SlJjIt8mj/4q9lxQ3Lom6ZpiKDs2NvIn9/bxCMLKlm+eTfpKUlMHz+EOaEiPj5yYEx3SAVFASBR4bGFVdz06GJ+fsVpXHJaQbe+9rY9Bz7swokc8Ndt38vBf979MlIjB/rcQwf7EXmZpGi+Piq5O0ur65lbXsmfFm2ivqGZgn59mR0KX5ROlxjpOAWARIXWVueS+95k254DzLtxaqc+abe2OpU79x12sF9aXc/WNpc0LujX99DB/uBUztDcPpqvj1ENTS28sGwLj5ZX8sbqbQCcNSqP2aFCLjx5yHEvapfoFAASNRas38Hs/3yb66eN4bppJcfctrG5lVVbdx+awlkWuaTx7gPh+frkJGN0ftahDpzw9XByyc3QnHG8qtq5j8cXbuLRhZVU7dxPTp8ULjmtgDmhIsYX5Cjk26EAkKjyzf99l3kVW5l30ycZmhtuC93d0ERFze7DPtWv2rqbppbwv8++qcmUDs0+rOVyzOBsffpLUK2tzjtrtzO3vJJnl2zmQHMr44ZkMydUxGcmFuikfRsKAIkqlTv2cf49rzGhMJf87HSWVtezYfuHlzQemJl2+FUuh4Uvaay7Ukl76vY38ZfF1TxaXsniqjpSky1yN7MiztHdzBQAEn0OXi10+ICMw341e/KwXAZlp+urvHTK8s0H72a2iR17GxmS04fLTi9g9ulFCXs3sy4HgJlNB34OJAO/cfd/Ocp2lwOPAme4e7mZpQK/ASYRvv3kf7n7TyLb9os8Nh5w4B/d/e1j1aEAiB/uzoHmVk3hSI9obG5l3vItzC2v4tUVW2l1mBy5m9nFCXY3sy4FgJklAyuBC4AqYAFwpbsvO2K7bOBpIA24NhIAVwGz3P0KM8sAlgFT3X29mT0I/NXdf2NmaUCGu+86Vi0KABE5UZvrGnhiURWPllexbttestJTmHlq+G5mk4bH/93MjhUAHYnBycBqd18bebGHgUsIH8zbugO4C7ipzToHMs0sBegLNAL1ZpYDnAP8PYC7N0YeExHpVkNy+3DN1NF845OjKN+wk0cWVPLn96p5eEElo/IzmRMq4tJJBQzKTryL0nXkVzAFQGWb5arIukPMbCJQ5O5PHfHcx4C9QA2wEbjb3XcAI4Fa4AEzW2RmvzGzdifozOxqMys3s/La2toODUpE5EhmxhnFA7h79gQW3DaNOy87hX4Zafzk2eV8/Cfz+MqD5bywdDNNLa1Bl9prOvINoL3vR4fmjcwsCbiXyKf5I0wGWoBhQH/gr2b2UuR9JwH/5O7zzeznwM3A9z/yRu73A/dDeAqoA/WKiBxTVnoKnztjOJ87Y/hhdzN7qWILeVlpfHZSIXNChYwelB10qT2qIwFQBRS1WS4EqtssZxM+kftqZC5tCPCkmc0CrgKec/cmYKuZvQmEgNeBKnefH3mNxwgHgIhIrxo9KItbLirlpk+NPXQ3s9+9sY77X18b93cz68gU0AKgxMxGRE7WXgE8efBBd69z9zx3L3b3YuAdwid+ywlP+5xnYZnAFGC5u28GKs1sbORlzuej5xRERHpNanIS08oGc/8XQ7x9y/n888Wl1Dc0c/MTHzD5Ry9z49zFvLN2O7HUOn88x/0G4O7NZnYt8DzhNtDfuftSM7sdKHf3J4/x9PuAB4AlhKeSHnD39yOP/RPwh0iorAX+oQvjEBHpNvnZ6Xz1nJF85ewRh93N7PF3qzhpYAZzQkV8dlLBoV+yxyr9EExEpAP2N7bw7JIa5pZX8s7a2LmbmX4JLCLSjTZsD9/N7LGFH97N7DMTwxelKx0aXXczUwCIiPSAllbnjdXbmFteyYtLt9DY0sopBbnMCRUya0JBVFyZVgEgItLDdh68m1l5FRU19aSlJDH95PDdzM4cFdzdzBQAIiK9aMmmOh4tr+RP71VTt7+Jgn59ufz08N3Migb07t3MFAAiIgFoaGrhxWVbmBu5m5k7nDV6IHNCRb12NzMFgIhIwDbt2s/jC6uYWx6+m1l2nxQuOW0Yc0JFnFKQ22MXpVMAiIhEidZW551125m74PC7mc0OFRvEil8AAAScSURBVPGZ04YxMCu9W99PASAiEoXau5vZtNLBzAkVcXZJHinJHblYw7EpAEREotyRdzMbnJPOZZMKmR0qYkQX7mamABARiRFHu5vZ/3z5Y6SlnPg3gq7eEEZERHpJWkoS08cPZfr4oWypb+Dxd6vYuH1fpw7+x6MAEBGJUoNzwncz6yndHykiIhITFAAiIglKASAikqAUACIiCUoBICKSoBQAIiIJSgEgIpKgFAAiIgkqpi4FYWa1wIZOPj0P2NaN5QQpXsYSL+MAjSUaxcs4oGtjOcnd89t7IKYCoCvMrPxo18OINfEylngZB2gs0ShexgE9NxZNAYmIJCgFgIhIgkqkALg/6AK6UbyMJV7GARpLNIqXcUAPjSVhzgGIiMjhEukbgIiItKEAEBFJUHEZAGb2OzPbamZL2qwbYGYvmtmqyH/7B1ljRxxlHP/XzDaZ2XuRPxcHWWNHmVmRmb1iZhVmttTMrousj6n9coxxxNx+MbM+ZvY3M1scGcv/i6wfYWbzI/vkETNLC7rW4znGWH5vZuva7JfTgq61I8ws2cwWmdlTkeUe2SdxGQDA74HpR6y7GXjZ3UuAlyPL0e73fHQcAPe6+2mRP8/0ck2d1Qzc6O6lwBTgm2ZWRuztl6ONA2JvvxwAznP3CcBpwHQzmwLcSXgsJcBO4MsB1thRRxsLwHfa7Jf3givxhFwHVLRZ7pF9EpcB4O6vAzuOWH0J8GDk7w8Cn+nVojrhKOOISe5e4+7vRv6+m/A/7gJibL8cYxwxx8P2RBZTI38cOA94LLI+6vcJHHMsMcfMCoEZwG8iy0YP7ZO4DICjGOzuNRD+nxgYFHA9XXGtmb0fmSKK6imT9phZMTARmE8M75cjxgExuF8iUw3vAVuBF4E1wC53b45sUkWMBNyRY3H3g/vlR5H9cq+ZpQdYYkf9DPgu0BpZHkgP7ZNECoB48UtgFOGvuTXAT4Mt58SYWRbwOPBtd68Pup7OamccMblf3L3F3U8DCoHJQGl7m/VuVZ1z5FjMbDxwCzAOOAMYAHwvwBKPy8xmAlvdfWHb1e1s2i37JJECYIuZDQWI/HdrwPV0irtvifxDbwV+Tfh/2phgZqmED5p/cPcnIqtjbr+0N45Y3i8A7r4LeJXweY1+ZpYSeagQqA6qrs5oM5bpkSk7d/cDwANE/345C5hlZuuBhwlP/fyMHtoniRQATwJfivz9S8CfA6yl0w4eLCMuBZYcbdtoEpnH/C1Q4e73tHkopvbL0cYRi/vFzPLNrF/k732BaYTPabwCXB7ZLOr3CRx1LMvbfLgwwvPmUb1f3P0Wdy9092LgCmCeu3+eHtoncflLYDN7CJhK+BKqW4D/A/wJmAsMBzYCs909qk+wHmUcUwlPMziwHvjawTn0aGZmnwD+CnzAh3ObtxKeP4+Z/XKMcVxJjO0XMzuV8AnFZMIfBue6++1mNpLwp88BwCLgC5FP0FHrGGOZB+QTnkZ5D/h6m5PFUc3MpgI3ufvMntoncRkAIiJyfIk0BSQiIm0oAEREEpQCQEQkQSkAREQSlAJARCRBKQBERBKUAkBEJEH9f+33TuF1pePGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "x_axis = min_child_samples_s\n",
    "plt.plot(x_axis,-test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.49032488, -0.48608582, -0.48654449, -0.48561977])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 3 candidates, totalling 9 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   4 out of   9 | elapsed:  7.7min remaining:  9.6min\n",
      "[Parallel(n_jobs=4)]: Done   6 out of   9 | elapsed: 14.7min remaining:  7.3min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48561976505170185\n",
      "{'min_child_samples': 40}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   9 out of   9 | elapsed: 16.8min finished\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.1,\n",
    "         'max_bin':127,\n",
    "         'subsample':0.7,\n",
    "         'bagging_freq':1,\n",
    "         'colsample_bytree':0.7,\n",
    "         'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75}\n",
    "min_child_samples_s = range(40,70,10)\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "tuned_parameters = dict(min_child_samples=min_child_samples_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='neg_log_loss',verbose=5,refit=False)\n",
    "grid_search.fit(x_train,y_train)\n",
    "\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3df5xU9X3v8debn+IKArKL/FJA+aUSBSdEYzTKGosmxSZXUrU/TNJckjbqba65rd60Saq3TWpMjbc1yTUqoTcmJuXWlpoEoxiikkgYGkVhF10RZQXZJSgoCsuPz/3jnN3MrgM7Czu7szPv5+Phgznf8/3OfM/xfM9753tm5igiMDMza9WvtztgZmalxcFgZmbtOBjMzKwdB4OZmbXjYDAzs3YG9HYHusOoUaNi4sSJvd0NM7M+Zc2aNdsjorpjeVkEw8SJE8lms73dDTOzPkXSS/nKPZVkZmbtOBjMzKwdB4OZmbXjYDAzs3YKCgZJ8yRtkNQg6cbD1LtCUkjKpMsDJS2W9IykOkk35dQdLmmJpPp03bkdnutz6XONOtKNMzOzruv0U0mS+gN3Ah8AGoHVkpZGxPoO9YYC1wOrcooXAIMjYqakY4H1kr4fEZuAO4BlEXGFpEHAsTnPNSF9vZePauvMzKzLCnnHMAdoiIiNEdEC3A9cnqfeLcCtwJ6csgCqJA0AhgAtwC5Jw4ALgHsAIqIlIl7PaXc78BdpezMz60GFBMM4YHPOcmNa1kbSLGBCRDzYoe0SYDewleSv/9siYgcwGWgGFkn6taS7JVWlzzUfeCUinj5cpyQtlJSVlG1ubi5gM8zMysP+AwdZvWkHX/5JHU279nTeoIsK+YKb8pS1/SUvqR/JX/gfy1NvDnAAGAuMAB6X9Ej6urOB6yJilaQ7gBslfRn4PHBJZ52KiLuAuwAymYzfWZhZWdu1Zx+PPdfM8romfrahidff2seAfmLOxJHUDjumW1+rkGBoBCbkLI8HtuQsDwXOAFZIAjgRWJr+5X81yXWEfUCTpJVABngMaIyI1usRS4AbgVOAScDT6XONB/5T0pyIePXINtHMrG/atH03j9Rt49H6Jn714g72HwxGHDuQudNqqJ0xmvOnjmLYMQO7/XULCYbVwBRJk4BXgCtJTvgARMROoO2TQ5JWAJ+LiKykWmCupO+SXFw+B/h6RLwqabOkaRGxAagF1kfEM0BNznNtAjIRsf0ot9PMrOTtP3CQNS+9xvL6JpbXbeOF5t0ATB19HJ88fzIXz6hh1kkj6N8v30RO9+k0GCJiv6RrgYeA/sC9EbFO0s1ANiKWHqb5ncAi4FmSKalFEbE2XXcdcF/6iaSNwMePYjvMzPqknW/v4+fPNbO8bhsrNjSz8+19DOwvzpl8An94zsnUTh/NSScc2/kTdSOVwz2fM5lM+Ef0zKyveHH7bpbXbeORum2s3vQaBw4GI6sGcdG0Gmpn1HD+lFEMLcIUUUeS1kREpmN5Wfy6qplZKdt/4CDZl15jed02ltc1sXF7MkU0bfRQPnXBZGpn1HDWhOJPERXKwWBmVgQ739rHiueaWF7XxIoNTezas59B/fvxnskjuea9E5k7vYYJI3t2iqhQDgYzs27yQvObbe8Ksi8lU0QnVA3iktNP5OIZNbxvSjXHDS79027p99DMrETtS79otryuiUfrm3gxnSKafuJQPv3+ydTOGM1Z44fTr0SmiArlYDAz64LX32phxYZmHqnbxs+fa+aNdIro3FNO4OPnJVNE40eU5hRRoRwMZmaHERG80Lw7Z4poBwcDRh03iEvPOJG500dz/pRRVPWBKaJClc+WmJl1k30HDrL6xR08UtfE8vptvPSbtwCYMWYYn7noVOZOr+HMPjhFVCgHg5kZ8NruFlY818QjdU08tqGZN/buZ9CAfrz3lBP45PmTmTu9hnHDh/R2N3uEg8HMKlJE0ND0ZtvPT6x56bV0imgwl80cQ+2MGt43ZRTHDqq802TlbbGZVayW/Qf51Ys72n6Y7uUdyRTR6WOHce1Fp1I7YzQzxx1ftlNEhXIwmFlZ+82be1mxoZnl9dt47LntvLl3P4MH9OO8U0fxqfcnU0Rjjq+MKaJCORjMrKxEBM9te5Pl9cmniP7z5deIgJqhg/ndM8dQO3005506iiGD+vd2V0uWg8HM+ry9+w+wauMOHq1v4pG6bTS+9jYAZ4wbxvVzp3DxjNGcPnZYxU8RFcrBYGZ90vY39/Kz+uQbx48918zulgMMHtCP9506ij+7MPlI6YnHd++dzSqFg8HM+oSIYMO2N1hel3yK6NebXycCRg8bzPyzxnHxjBree4qniLqDg8HMStbe/Qd4cuOOtm8dv/J6MkX0rvHH8+e1U6mdUcPpY4eR3grYuklBwSBpHnAHyR3c7o6Irxyi3hXAvwDvTm/tORC4G5idvtY/R8SX07rD03VnAAF8IiJ+KekW4HLgINAEfCwitrzz1cysHDW/sZefbUjeFTz+/HbeajnAMQP78b5Tq7lu7qlcNL2G0cM8RVRMnQaDpP4kt+j8ANAIrJa0NCLWd6g3FLgeWJVTvAAYHBEzJR0LrJf0/YjYRBI0yyLiivT2nq2/OvXViPjr9DmvB74AfPpoNtLMSldEULf1DR6t38YjdU083ZhMEY05/hg+PGscF88YzbmnnMAxAz1F1FMKeccwB2iIiI0Aku4n+Yt+fYd6twC3Ap/LKQugStIAYAjQAuySNAy4APgYQES0pOuIiF057avS5zCzMrJn3wF+ufE3PJr+XHXrFNGZ44/nsxcnU0SnjfEUUW8pJBjGAZtzlhuB9+RWkDQLmBARD0rKDYYlJCGyleQdwWcjYoeks4BmYJGkM4E1wH+LiN3p8/0t8MfATuCifJ2StBBYCHDSSScVsBlm1pua3tjDz+qTO5o90ZBMEQ0Z2J/3TRnF9bXJFFHNUE8RlYJCgiFfZLf9FS+pH3A76V//HcwBDgBjgRHA45IeSV93NnBdRKySdAdwI/DXABHxeeDzkm4CrgW++I4ORNwF3AWQyWT8rsKsxEQE67fuSj5FVN/E05tfB2Ds8cfwkdnjqJ0xmnMne4qoFBUSDI3AhJzl8UDuxeChJBeQV6Rv+04ElkqaD1xNch1hH9AkaSWQAR4DGiOi9XrEEpJg6Oh7wI/IEwxmVnr27DvAL1/4TdtvEW3duQcJzhw/nM9dMpW500czY8xQTxGVuEKCYTUwRdIk4BXgSpITPgARsRMY1bosaQXwufRTSbXAXEnfJZlKOgf4ekS8KmmzpGkRsQGoJb1mIWlKRDyfPt18oP5oN9LMiqdp1570G8dNrGzYztv7DnDsoP6cP2UUn/3AVC6aVkP10MG93U3rgk6DISL2S7oWeIjk46r3RsQ6STcD2YhYepjmdwKLgGdJpqQWRcTadN11wH3pJ5I2Ah9Py78iaRrJx1Vfwp9IMispEcG6La1TRNtY27gTgHHDh7AgM57aGaN5z6SRniLqwxTR96fnM5lMZLPZ3u6GWdnas+8AKxu2s7y+iUfrmnh1VzJFNGvCcGpnjKZ2Rg3TRnuKqK+RtCYiMh3L/c1nM8tr2649LK9r4tH6bTzRsJ09+w5SNag/F0ytZu70Gi6aXsOo4zxFVI4cDGYGJFNEz76yq+3C8TOvJFNE40cM4cp3n8Tc6TW8Z/JIBg/wFFG5czCYVbC3W1qniJIw2LZrLxLMPmkEfzFvGrXTRzN19HGeIqowDgazCrN159s8mn7RbGXDdvbuP8hxgwdwwdRR1E4fzYXTqjnBU0QVzcFgVuYOHgyeeWVn203v121JfnVmwsghXDXnJC6eMZo5k0YyaEC/Xu6plQoHg1kZeqtlP088vz25eLyhieY39tJPcPbJI/jLedO5eEYNp9Z4isjyczCYlYktr7/d9q7gFy/8hpb9Bxk6eAAXTKumdnoNF06rYWTVoN7upvUBDgazPurgwWDtKztZXpf8XHXd1mSK6OQTjuUP33MytTNqePdETxFZ1zkYzPqQ3Xv380TDdpbXbePR+ma2v5lMEWVOHslNl06ndkYNp1R7isiOjoPBrMS98vrbbbe2/OXGdIromAG8f2o1F89IPkU0/FhPEVn3cTCYlZiDB4OnGl9vC4P6V98AYOIJx/JH5/x2imhgf08RWXE4GMxKwJt79/PE8808UtfEig1NbH+zhf79RObkEfzPy6ZTO2M0p1Qf19vdtArhYDDrJY2vvcXyuiYeqdvGqo07aDlwkGHHDODCaTXUzqjh/VM9RWS9w8Fg1kMOHAye2vzbKaIN25IposmjqrjmvSdTO2M0Z588wlNE1uscDGZF9MaefTzx/Pa2KaLf7E6miN49cQR/9cEZzJ1ew2RPEVmJcTCYFcl3n3yJm/9jPS0HDnL8kIFcOK2a2hmjef/Uao4fMrC3u2d2SA4GsyL4519u4gv/vo73T63mzy48hbNPHsEATxFZH1HQkSppnqQNkhok3XiYeldICkmZdHmgpMWSnpFUJ+mmnLrDJS2RVJ+uOzct/2patlbSA5KGH+1GmvWk1lC4eMZovv3HGd4z+QSHgvUpnR6tkvqT3Lv5UuA04CpJp+WpNxS4HliVU7wAGBwRM4GzgU9JmpiuuwNYFhHTgTOBurT8YeCMiHgX8BxwE2Z9RG4ofOMPZvvnKKxPKuSonQM0RMTGiGgB7gcuz1PvFuBWYE9OWQBVkgYAQ4AWYJekYcAFwD0AEdESEa+nj38aEfvT9k8C47u+WWY9z6Fg5aKQI3ccsDlnuTEtayNpFjAhIh7s0HYJsBvYCrwM3BYRO4DJQDOwSNKvJd0tqSrPa38C+Em+TklaKCkrKdvc3FzAZpgVj0PBykkhR2++X+OKtpVSP+B24IY89eYAB4CxwCTgBkmTSS56zwa+GRGzSMKj3bULSZ8H9gP35etURNwVEZmIyFRXVxewGWbF4VCwclPIEdwITMhZHg9syVkeCpwBrJC0CTgHWJpegL6a5DrCvohoAlYCmfQ5GyOi9XrEEpKgAEDSNcCHgD+IiLYQMis1DgUrR4UcxauBKZImSRoEXAksbV0ZETsjYlRETIyIiSTXBeZHRJZk+miuElUkoVEfEa8CmyVNS5+mFlgPySeggL9Mn+Ot7tlMs+7nULBy1en3GCJiv6RrgYeA/sC9EbFO0s1ANiKWHqb5ncAi4FmSKalFEbE2XXcdcF8aNhuBj6fl/wQMBh5Of1P+yYj4dNc3zax4WkPhA6eN5s6rHQpWXlQOMzWZTCay2Wxvd8MqhEPByoWkNRGR6VjuI9qsCxwKVgl8VJsVyKFglcJHtlkBHApWSXx0m3XCoWCVxke42WE4FKwS+Sg3OwSHglUqH+lmeTgUrJL5aDfrYPEvklC4xKFgFcpHvFmOxb/YxBeXJqHwTw4Fq1A+6s1SDgWzhI98MxwKZrl89FvFcyiYtecRYBXNoWD2Th4FVrEcCmb5eSRYRXIomB2aR4NVnNZQ+J3THQpm+RQ0IiTNk7RBUoOkGw9T7wpJkd7vGUkDJS2W9IykOkk35dQdLmmJpPp03blp+QJJ6yQdbH0es+7ynZUvtoXCP17lUDDLp9NRIak/yS06LwVOA66SdFqeekOB64FVOcULgMERMRM4G/iUpInpujuAZRExHTgTqEvLnwU+Ajx2BNtjdkjfWfkiX/qP9Q4Fs04UMjLmAA0RsTEiWoD7gcvz1LsFuBXYk1MWQJWkAcAQoAXYJWkYcAFwD0BEtETE6+njuojYcKQbZJaPQ8GscIWMjnHA5pzlxrSsjaRZwISIeLBD2yXAbmAr8DJwW0TsACYDzcAiSb+WdLekqq50XNJCSVlJ2ebm5q40tQrjUDDrmkJGiPKURdtKqR9wO3BDnnpzgAPAWGAScIOkycAAYDbwzYiYRRIeh7x2kU9E3BURmYjIVFdXd6WpVRCHglnXFTJKGoEJOcvjgS05y0OBM4AVkjYB5wBL0wvHV5NcR9gXEU3ASiCTPmdjRLRej1hCEhRm3cahYHZkChkpq4EpkiZJGgRcCSxtXRkROyNiVERMjIiJwJPA/IjIkkwfzVWiiiQ06iPiVWCzpGnp09QC67tvs6zSORTMjlynoyUi9gPXAg+RfHLohxGxTtLNkuZ30vxO4DiSTxqtBhZFxNp03XXAfZLWAmcBfwcg6cOSGoFzgR9JeugItssqmEPB7OgoIjqvVeIymUxks9ne7oaVgNxQ+KerZzOwv0PB7FAkrYmId3xfbEBvdMasGBatfJG/cSiYHTWPHCsLDgWz7uPRY32eQ8Gse3kEWZ/mUDDrfh5F1mc5FMyKwyPJ+iSHglnxeDRZn+NQMCsujyjrU1pDYd7pJzoUzIrEo8r6jNxQ+MerZzkUzIrEX3CzPuHeJ17k5gcdCmY9waPLSp5DwaxneYRZSXMomPU8jzIrWQ4Fs97hkWYlyaFg1ns82qzkOBTMepdHnJUUh4JZ7/Oos5LRGgqXnuFQMOtNBY08SfMkbZDUIOnGw9S7QlJIyqTLAyUtlvSMpDpJN+XUHS5piaT6dN25aflISQ9Lej79d8TRbqSVvtxQ+N9XORTMelOno09Sf5J7N18KnAZcJem0PPWGAtcDq3KKFwCDI2ImcDbwKUkT03V3AMsiYjpwJsn9pAFuBJZHxBRgebpsZeweh4JZSSlkBM4BGiJiY0S0APcDl+epdwtwK7AnpyyAKkkDgCFAC7BL0jDgAuAegIhoiYjX0zaXA4vTx4uB3+vaJllfcs8TL3KLQ8GspBQyCscBm3OWG9OyNpJmARMi4sEObZcAu4GtwMvAbRGxA5gMNAOLJP1a0t2SqtI2oyNiK0D6b02+TklaKCkrKdvc3FzAZlipcSiYlaZCRqLylEXbSqkfcDtwQ556c4ADwFhgEnCDpMkkv9E0G/hmRMwiCY8uTRlFxF0RkYmITHV1dVeaWglwKJiVrkJGYyMwIWd5PLAlZ3kocAawQtIm4BxgaXoB+mqS6wj7IqIJWAlk0udsjIjW6xFLSIICYJukMQDpv01HsmFWuhwKZqWtkBG5GpgiaZKkQcCVwNLWlRGxMyJGRcTEiJgIPAnMj4gsyfTRXCWqSEKjPiJeBTZLmpY+TS2wPn28FLgmfXwN8O9Ht4lWShwKZqWv05/djoj9kq4FHgL6A/dGxDpJNwPZiFh6mOZ3AouAZ0mmpBZFxNp03XXAfWnYbAQ+npZ/BfihpD8hCZYFR7BdVoJaQ+GymSdyx5UOBbNSpYjovFaJy2Qykc1me7sbdhgOBbPSI2lNRGQ6lnt0WtE5FMz6Ft/BzYrq7sc38r9+VOdQMOtDPEqtaBwKZn2TR6oVhUPBrO/yaLVu51Aw69s8Yq1bORTM+j6PWus2DgWz8uCRa92iNRQ+OHOMQ8Gsj/PotaOWGwpfv/Ish4JZH+cRbEfFoWBWfjyK7Yg5FMzKk0eyHRGHgln58mi2LnMomJU3j2jrEoeCWfnzqLaCORTMKoNHthXEoWBWOQoa3ZLmSdogqUHSjYepd4WkSO/3jKSBkhZLekZSnaSbcupuSsufkpTNKT9T0i/Tdf8hadjRbKAdvfZfXnMomJW7Tke4pP4kt+i8FDgNuErSaXnqDQWuB1blFC8ABkfETOBs4FOSJuasvygizupwB6G7gRvTNg8A/6NLW2TdqmMoDHAomJW9Qkb5HKAhIjZGRAtwP3B5nnq3ALcCe3LKAqiSNAAYArQAuzp5vWnAY+njh4H/UkAfrQgcCmaVqZCRPg7YnLPcmJa1kTQLmBARD3ZouwTYDWwFXgZui4gd6boAfippjaSFOW2eBeanjxcAE/J1StJCSVlJ2ebm5gI2w7ri2485FMwqVSGjXXnKom2l1A+4HbghT705wAFgLDAJuEHS5HTdeRExm2SK6jOSLkjLP5EurwGGkrzLeGcHIu6KiExEZKqrqwvYDCvUtx/byN/+2KFgVqkKuedzI+3/ah8PbMlZHgqcAayQBHAisFTSfOBqYFlE7AOaJK0EMsDGiNgCEBFNkh4gCZHHIqIeuARA0lTgg0exfdZFDgUzK2TUrwamSJokaRBwJbC0dWVE7IyIURExMSImAk8C8yMiSzJ9NFeJKuAcoF5SVXqxmrT8EpIpJCTVpP/2A/4K+FY3bat1wqFgZlBAMETEfuBa4CGgDvhhRKyTdHP6ruBw7gSOIznprwYWRcRaYDTwhKSngV8BP4qIZWmbqyQ9B9STvDNZdATbZV3kUDCzVoqIzmuVuEwmE9lstvOKlldbKLxrDHf8vkPBrFJIWtPh6wKAv/lc8RwKZtaRzwIVzKFgZvn4TFChHApmdiiFfFzVysxdj73A3/243qFgZnn5jFBhHApm1hmfFSqIQ8HMCuEzQ4VwKJhZoXx2qACtofAhh4KZFcBniDKXGwpfdyiYWQF8lihjDgUzOxI+U5Qph4KZHSmfLcqQQ8HMjoa/4FZm/s/PX+DLP3EomNmR81mjjDgUzKw7+MxRJhwKZtZdfPYoAw4FM+tOvsbQx7WGwu+eOZbbP3qmQ8HMjlpBZxFJ8yRtkNQg6cbD1LtCUkjKpMsDJS2W9IykOkk35dTdlJY/JSmbU36WpCdbyyXNOZoNLGcOBTMrhk7fMUjqT3Lv5g8AjcBqSUsjYn2HekOB64FVOcULgMERMVPSscB6Sd+PiE3p+osiYnuHl7wV+JuI+Imky9LlC7u+aeXNoWBmxVLI2WQO0BARGyOiBbgfuDxPvVtITuJ7csoCqJI0ABgCtAC7Onm9AIalj48HthTQx4riUDCzYirkjDIO2Jyz3JiWtZE0C5gQEQ92aLsE2A1sBV4GbouIHem6AH4qaY2khTlt/hz4qqTNwG3ATeQhaWE61ZRtbm4uYDPKg0PBzIqtkLOK8pRF20qpH3A7cEOeenOAA8BYYBJwg6TJ6brzImI2cCnwGUkXpOV/Cnw2IiYAnwXuydepiLgrIjIRkamuri5gM/q+bzkUzKwHFHJmaQQm5CyPp/30zlDgDGCFpE3AOcDS9AL01cCyiNgXEU3ASiADEBFb0n+bgAdIQgTgGuBf08f/klNe0b718xf4ikPBzHpAIWeX1cAUSZMkDQKuBJa2royInRExKiImRsRE4ElgfkRkSaaP5ipRRRIa9ZKq0ovVpOWXAM+mT7kFeH/6eC7w/FFvZR/nUDCzntTpp5IiYr+ka4GHgP7AvRGxTtLNQDYilh6m+Z3AIpKTvoBFEbE2nU56QFJrH74XEcvSNv8VuCO9YL0HWPjOp60craEw/8yx/INDwcx6gCKi81olLpPJRDab7bxiH+NQMLNikrQmIjIdy32mKVEOBTPrLT7blCCHgpn1Jp9xSoxDwcx6m886JcShYGalwL+uWiK+ueIF/n6ZQ8HMep/PPiXAoWBmpcRnoF7mUDCzUuOzUC9qDYXLz3IomFnp8Jmol+SGwtcWOBTMrHT4bNQLHApmVsp8RuphDgUzK3U+K/Ugh4KZ9QX+HkMP+caKBm5dtsGhYGYlz2enHuBQMLO+xGeoInMomFlf47NUETkUzKwvKuhMJWmepA2SGiTdeJh6V0iK9H7PSBooabGkZyTVSbopp+6mtPwpSdmc8h+kZU+ldZ46mg3sLbmh8A8fPcuhYGZ9RqcXnyX1J7lF5weARmC1pKURsb5DvaHA9cCqnOIFwOCImCnpWGC9pO9HxKZ0/UURsT33eSLi93Oe82vAzq5vVu/qGAr9+6m3u2RmVrBC/oydAzRExMaIaAHuBy7PU+8W4FaS+zS3CqAqvX/zEKAF2FVIx5TcEPqjwPcLqV8qHApm1tcVEgzjgM05y41pWRtJs4AJEfFgh7ZLgN3AVuBl4LaI2JGuC+CnktZIWpjndc8HtkXE8/k6JWmhpKykbHNzcwGbUXwOBTMrB4UEQ76zW7StlPoBtwM35Kk3BzgAjAUmATdImpyuOy8iZgOXAp+RdEGHtldxmHcLEXFXRGQiIlNdXV3AZhSXQ8HMykUhwdAITMhZHg9syVkeCpwBrJC0CTgHWJpegL4aWBYR+yKiCVgJZAAiYkv6bxPwAEmIAJBOPX0E+MGRbVbPuvNnDgUzKx+FBMNqYIqkSZIGAVcCS1tXRsTOiBgVERMjYiLwJDA/IrIk00dzlagiCY16SVXpxWrS8kuAZ3Ne82KgPiIau2Ebi+rOnzXw1YccCmZWPjoNhojYD1wLPATUAT+MiHWSbpY0v5PmdwLHkZz0VwOLImItMBp4QtLTwK+AH0XEspx2V9IHLjo7FMysHCkiOq9V4jKZTGSz2c4rdqPWUPi9s8byNYeCmfVBktZERKZjub91dQQcCmZWzhwMXeRQMLNy52DoAoeCmVUCB0OBHApmVikcDAVwKJhZJXEwdMKhYGaVxsFwGA4FM6tEvufzIbSGwodnjeO2BWc6FMysYvgdQx4OBTOrZA6GDhwKZlbpHAw5HApmZg6GNg4FM7OEgwGHgplZrooPBoeCmVl7FR0M31jhUDAz66iig2HiCVUsOHu8Q8HMLEdFf8HtspljuGzmmN7uhplZSSnoHYOkeZI2SGqQdONh6l0hKSRl0uWBkhZLekZSnaSbcupuSsufkpTt8DzXpa+3TtKtR7pxZmbWdZ2+Y5DUn+TezR8AGoHVkpZGxPoO9YYC1wOrcooXAIMjYqakY4H1kr4fEZvS9RdFxPYOz3MRcDnwrojYK6nmCLfNzMyOQCHvGOYADRGxMSJagPtJTtwd3QLcCuzJKQugStIAYAjQAuzq5PX+FPhKROwFiIimAvpoZmbdpJBgGAdszlluTMvaSJoFTIiIBzu0XQLsBrYCLwO3RcSOdF0AP5W0RtLCnDZTgfMlrZL0c0nvztcpSQslZSVlm5ubC9gMMzMrRCEXn/N9XCfaVkr9gNuBj+WpNwc4AIwFRgCPS3okIjYC50XElnSq6GFJ9RHxWNqnEcA5wLuBH0qaHBGR+8QRcRdwF0Amk2m3zszMjlwh7xgagQk5y+OBLTnLQ4EzgBWSNpGc0JemF6CvBpZFxL50SmglkAGIiC3pv03AAyQh0vp6/xqJXwEHgVFHtnlmZtZVhQTDamCKpEmSBgFXAktbV0bEzogYFRETI2Ii8CQwPyKyJNNHc5WoIgmNeklV6cVq0uG9WzEAAAX8SURBVPJLgGfTp/w3YG66biowCGh3gdrMzIqn06mkiNgv6VrgIaA/cG9ErJN0M5CNiKWHaX4nsIjkpC9gUUSslTQZeEBSax++FxHL0jb3AvdKepbkYvU1HaeRzMyseFQO51xJzcBLR9h8FKX5jsT96hr3q2vcr64p1X7B0fXt5Iio7lhYFsFwNCRlIyLT2/3oyP3qGvera9yvrinVfkFx+lbRv5VkZmbv5GAwM7N2HAzpdyFKkPvVNe5X17hfXVOq/YIi9K3irzGYmVl7fsdgZmbtOBjMzKydsg8GSf0l/VrSg+nypPQH+p6X9IP029z52t2U3n9ig6Tf6YF+3Ze+1rOS7pU08BDtDqT3sHhK0uG+XNhd/fqOpBdzXvOsQ7S7Jt2nz0u6pgf69XhOn7ZI+rdDtCva/sp3TxFJIyU9nO6HhyWNOETbou2vQ/Trq5LqJa2V9ICk4YW2LXK/viTplZz/R5cdom1B94Tpxn79IKdPmyQ9VWjbbuzXcElL0v9vdZLO7bHjKyLK+j/gvwPfAx5Ml38IXJk+/hbwp3nanAY8DQwGJgEvAP2L3K/LSL4dLuD7+fqV1nuzh/fXd4ArOmkzEtiY/jsifTyimP3qsO7/AX/c0/sL2ASM6lB2K3Bj+vhG4O97en8dol+XAAPSx3+fr1+Halvkfn0J+Fwn7fqnY3AyyU/kPA2cVsx+dVj/NeALvbC/FgOfTB8PAob31PFV1u8YJI0HPgjcnS6L5HeYlqRVFgO/l6fp5cD9EbE3Il4EGvjtj/x1e78AIuLHkQJ+RfJjhT0qX78K9DvAwxGxIyJeAx4G5vVEv5T85tZckt/YKgWXkxxXcOjjq6j7K5+I+GlE7E8Xn6QXjq+jUOg9Ybpdes74KMkfaz1G0jDgAuAegIhoiYjX6aHjq6yDAfg68Bckv9AKcALwes4Aece9JVKd3oOim/vVJp1C+iNgWcd1qWOU3IfiSUn5Dopi9Otv0ymI2yUNztOu1/YX8GFgeUQc6gZQxdxf+e4pMjoitgKk/+a7A2Gx99eh7nXS6hPAT46wbTH6dW16fN17iKmR3txf5wPbIuL5I2h7NCYDzcCidAr1biU/ONojx1fZBoOkDwFNEbEmtzhP1Xyf1y20Xnf1K9c3gMci4vFDrD8pkq+/Xw18XdIpRe7XTcB0kntjjAT+Ml/zPGU9tb+u4vB/zRVlf6XOi4jZwKXAZyRdUGC7ou2v1CH7JenzwH7gvq62LVK/vgmcApxFckOvr+Vp12v7i86Pr2LtrwHAbOCbETGL5IZnhV5bOer9VbbBAJwHzFdyj4j7SaYbvg4MV3KrUXjnvSVadXYPim7tl6TvAkj6IlBNMp+eV/z2PhYbgRXArGL2KyK2pjNce0l+KTfflFpv7a8T0v786FCNi7i/cp87954i2ySNSfs3Bsh3a9pi7q9D9Yv0IuSHgD9IpywLblusfkXEtog4EBEHgW8f4vV6a38NAD4C/KCrbbtBI9AYEavS5SUkQdEzx1cxLpqU2n/Ahfz2Yuq/0P7i85/lqX867S8+b6SbLz7n6dcngV8AQw5TfwQwOH08CniebrwId4h+jUn/FUmwfiVP/ZHAi2n/RqSPRxazX+nyp4HFvbG/gCpgaM7jX5DM436V9hcHb+3J/XWYfs0D1gPVXW1b5H6NyanzWZJrex3bDkjH4CR+e/H59GL2K12eB/y8N/ZX+pyPA9PSx19Kj60eOb66ZQNK/b8OJ7rJJBd3G0hCovXEMR+4OafN50k+CbEBuLQH+rU/fb2n0v++kJZngLvTx+8FnkkHxjPAn/RAvx5NX+tZ4LvAcR37lS5/It2nDcDHi92vdHlFx4HYU/srPY6eTv9bB3w+LT8BWE4SQstbB2RP7a/D9KuBZN659fj6Vlo+Fvjx4doWuV//N/1/s5bkBmBjOvYrXb4MeC4dI0XvV7ruO8CnO9Tvkf2VPv9ZQDbdN/9GcpLvkePLP4lhZmbtlPM1BjMzOwIOBjMza8fBYGZm7TgYzMysHQeDmZm142AwM7N2HAxmZtbO/wfjEtTCx7D2DAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "x_axis = min_child_samples_s\n",
    "plt.plot(x_axis,-test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed: 24.4min remaining:  6.1min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.48561976505170185\n",
      "{'subsample': 0.7}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 30.5min finished\n"
     ]
    }
   ],
   "source": [
    "# 行采样参数 sub_samples/bagging_fraction  每行的采样比例\n",
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.1,\n",
    "         'max_bin':127,\n",
    "         'bagging_freq':1,\n",
    "         'colsample_bytree':0.7,\n",
    "         'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75,\n",
    "         'min_child_samples':40}\n",
    "subsamples_s = [i/10.0 for i in range(5,10)]\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "tuned_parameters = dict(subsample=subsamples_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='neg_log_loss',verbose=5,refit=False)\n",
    "grid_search.fit(x_train,y_train)\n",
    "\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3wUdf7H8dcnnYQQWugllABCkARWxAanWMBTQAUFrHeecP5ELOgdnlesp3IoluNUUDmwgIoNOcR2isoJsoEQSiihCAGE0EIJIYXP748dcBNDsoEkk/J5Ph55ZOc7882+Z8T97Mzsfr+iqhhjjDHHBbkdwBhjTNVihcEYY0whVhiMMcYUYoXBGGNMIVYYjDHGFBLidoDy0LhxY42Li3M7hjHGVCvJycm7VTW2aHuNKAxxcXF4vV63YxhjTLUiIj8W126XkowxxhRihcEYY0whVhiMMcYUYoXBGGNMIVYYjDHGFGKFwRhjTCFWGIwxxhRSqwvDsi37ePHrDW7HMMaYKqVWF4YPl23jqflr+Chlm9tRjDGmygioMIjIABFZKyLpIjK+hO2GioiKiMdZDhWR6SKyQkTSROQBv23vEpGVIrJKRO72a39IRLaJSIrzc/np7GBJHvx1V3rHNeQPs1NZkZFVUU9jjDHVSqmFQUSCgcnAQKArMEJEuhazXTQwFljs1zwMCFfV7kAvYLSIxIlIAnAb0BvoAVwhIvF+/SapaqLzM+8U961UYSFB/OuGnjSuG86o171kHjxaUU9ljDHVRiBnDL2BdFXdqKq5wCxgcDHbPQpMAHL82hSIEpEQoA6QCxwAzgAWqWq2quYDC4CrTn03Tl3juuG8fGMv9mXncvsbyRzNL3AjhjHGVBmBFIaWwFa/5Qyn7QQRSQJaq+rcIn1nA4eBHcAWYKKq7gVWAn1FpJGIRAKXA639+o0RkVQReU1EGhQXSkRGiYhXRLyZmZkB7MbJJbSM4R9De+D9cR9/+2gVNg+2MaY2C6QwSDFtJ145RSQImASMK2a73kAB0AJoB4wTkfaqmgY8BXwOzAeWA/lOnxeBDkAivoLydHGhVHWKqnpU1RMb+4tRY8vsyh4tuOPCDsxaspXXFxU74KAxxtQKgRSGDAq/m28FbPdbjgYSgK9FZDPQB5jj3IAeCcxX1TxV3QUsBDwAqvqqqvZU1b7AXmC9075TVQtU9RgwFV9xqRTjLulM/y5NePjj1fxvw+7KelpjjKlSAikMS4B4EWknImHAcGDO8ZWqmqWqjVU1TlXjgEXAIFX14rt8dJH4ROErGmsARKSJ87sNcDUw01lu7vfcV+G77FQpgoKEZ4cn0q5xFHe8uZSte7Mr66mNMabKKLUwODeHxwCfAmnAO6q6SkQeEZFBpXSfDNTF9+K+BJimqqnOuvdEZDXwMXCHqu5z2ic4H29NBS4E7inzXp2G6IhQpt7koeCYctsML4eP5pfeyRhjahCpCTdaPR6PlvcMbgvWZfKbaT9wWbdmTB7Zk6Cg4m61GGNM9SUiyarqKdpeq7/5XJJ+nWJ5YOAZfLLyJ174b7rbcYwxptJYYSjB7y5ox9VJLZn0xTo+XfWT23GMMaZSWGEogYjw96u706NVDPe+ncLanw66HckYYyqcFYZSRIQG8/KNHiLDQ/jdjCXsO5zrdiRjjKlQVhgC0Cwmgpdv7MXOrKOMmbmU/IJjbkcyxpgKY4UhQD3bNOCxqxJYmL6Hx+eluR3HGGMqTIjbAaqTaz2tSdtxgGkLN3NG83pc62ldeidjjKlm7IyhjB68/AzO79iYP3+wkuQf95XewRhjqhkrDGUUEhzEP0cm0Swmgt+/kcxPWTmldzLGmGrECsMpqB8Zxis3e8g+ms+o173k5NkcDsaYmsMKwynq1DSaSdclkpqRxQPvr7A5HIwxNYYVhtNwabdm3HtJJz5Yto1Xvt3kdhxjjCkXVhhO050XdeTy7s144pM0Fqw7vZnkjDGmKrDCcJpEhH8M7UGnptGMeWspGzMPuR3JGGNOixWGchAVHsLUmzyEBAm3zfByICfP7UjGGHPKrDCUk9YNI/nX9b34cU82d89KoeCY3Yw2xlRPARUGERkgImtFJF1Expew3VARUWe+Z0QkVESmOzOypYnIA37b3iUiK0VklYjc7dfeUEQ+F5H1zu8Gp7ODlemcDo3425Vd+e+aXTz92Vq34xhjzCkptTCISDC+KToHAl2BESLStZjtooGxwGK/5mFAuKp2B3oBo0UkTkQSgNuA3kAP4AoRiXf6jAe+VNV44Etnudq4oU9bRvRuzb++3sDHy7e7HccYY8oskDOG3kC6qm5U1VxgFjC4mO0eBSYA/l8FViBKREKAOkAucAA4A1ikqtnOnNILgKucPoOB6c7j6cCQsu2Su0SEhwcl4GnbgPtnL2fltiy3IxljTJkEUhhaAlv9ljOcthNEJAlorapzi/SdDRwGdgBbgImquhdYCfQVkUYiEglcDhwfka6pqu4AcH43KS6UiIwSEa+IeDMzq9bHRMNCgnjxhl40jAxj1Awvuw8ddTuSMcYELJDCIMW0nbizKiJBwCRgXDHb9QYKgBZAO2CciLRX1TTgKeBzYD6wHMgvS3BVnaKqHlX1xMbGlqVrpYiNDmfKTR72HM7l9jeSyc23ORyMMdVDIIUhg5/fzQO0AvwvnkcDCcDXIrIZ6APMcW5AjwTmq2qequ4CFgIeAFV9VVV7qmpfYC+w3vl7O0WkOYDze9ep7pzbElrGMGHomSzZvI+HPl7ldhxjjAlIIIVhCRAvIu1EJAwYDsw5vlJVs1S1sarGqWocsAgYpKpefJePLhKfKHxFYw2AiDRxfrcBrgZmOn9yDnCz8/hm4KPT3EdXDU5sye2/6sBbi7fw+qIf3Y5jjDGlKnWiHlXNF5ExwKdAMPCaqq4SkUcAr6rOKaH7ZGAavnsKAkxT1VRn3Xsi0gjIA+5Q1eOTGzwJvCMit+IrLMNOZceqkvsu7czanw7y8JxVxDepS5/2jdyOZIwxJyU1YVRQj8ejXq/X7RglOpCTx1WTF7IvO4+P7jiP1g0j3Y5kjKnlRCRZVT1F2+2bz5WkXkQoU2/ykFdwjNtmeMnOLdO9dmOMqTRWGCpR+9i6vDAiiXU7D3Lfu8ttDgdjTJVkhaGS/apzE8YP7MK8FT/xz/+mux3HGGN+odSbz6b83XZBe9J2HOTpz9fRuVk0l3Zr5nYkY4w5wc4YXCAiPHF1d85sFcM9b6ewbudBtyMZY8wJVhhcEhEazMs39qJOWAi3zfCyPzvX7UjGGANYYXBV85g6vHxjL3bsz2HMW8vIL7BhM4wx7rPC4LJebRvw2JAEvkvfzROfrHE7jjHG2M3nquDas1qzescBXv1uE2c0r8fQXq3cjmSMqcXsjKGKePDXZ3Buh0b86f0VLNuyr/QOxhhTQawwVBGhwUFMHtmTpjHhjH49mZ0HckrvZIwxFcAKQxXSICqMqTd5OHQ0n1GvJ5OTV+B2JGNMLWSFoYrp0qwez1ybyPKt+/nTByts2AxjTKWzwlAFDUhoxt0Xx/P+0m28+t0mt+MYY2oZKwxV1NiL4hmY0Iy/z0vjm3VVa05rY0zNZoWhigoKEiYO60GnptGMeWspm3cfdjuSMaaWCKgwiMgAEVkrIukiMr6E7YaKiDrzPSMioSIyXURWiEiaiDzgt+09IrJKRFaKyEwRiXDa/y0im0QkxflJPN2drK6iwkOYepOH4CDhdzO8HMzJczuSMaYWKLUwiEgwvik6BwJdgREi0rWY7aKBscBiv+ZhQLiqdgd6AaNFJE5EWjrbelQ1Ad+UocP9+t2vqonOT8op7luN0LphJJOv78mm3Ye55+0Ujh2zm9HGmIoVyBlDbyBdVTeqai4wCxhczHaPAhMA/w/gKxAlIiFAHSAXOOCsCwHqOOsige2ntgs137kdGvPXK7ryRdounvl8ndtxjDE1XCCFoSWw1W85w2k7QUSSgNaqOrdI39nAYWAHsAWYqKp7VXUbMNFp2wFkqepnfv0eF5FUEZkkIuHFhRKRUSLiFRFvZmbNvzl70zltGX5Wa/75VTpzU62GGmMqTiCFQYppO3E9Q0SCgEnAuGK26w0UAC2AdsA4EWkvIg3wnXW0c9ZFicgNTp8HgC7AWUBD4I/FhVLVKarqUVVPbGxsALtRvYkIDw/uRq+2Dbj/3VRWbc9yO5IxpoYKpDBkAK39lltR+LJPNJAAfC0im4E+wBznBvRIYL6q5qnqLmAh4AEuBjapaqaq5gHvA+cCqOoO9TkKTMNXXAwQHhLMSzf0on5kKKNmJLPn0FG3IxljaqBACsMSIF5E2olIGL6bxHOOr1TVLFVtrKpxqhoHLAIGqaoX36Wii8QnCl/RWOO09xGRSBERoD+QBiAizZ3fAgwBVpbTvtYIsdHhTLnRw+5DR7n9zaXk5tscDsaY8lVqYVDVfGAM8Cm+F+93VHWViDwiIoNK6T4ZqIvvxX0JME1VU1V1Mb77D0uBFU6OKU6fN0VkhdPeGHis7LtVs3VvFcOEoWfyw6a9PDJ3ldtxjDE1jNSEsXg8Ho96vV63Y1S6Jz5J4+UFG3n8qgSuP7ut23GMMdWMiCSrqqdou33zuRr7w2Vd+FXnWP720Sp+2LTX7TjGmBrCCkM1FhwkPDc8iTaNIrn9jWQy9mW7HckYUwNYYajmYuqEMvUmD7n5xxg1I5kjuTaHgzHm9FhhqAE6xNbl+RFJpP10gPtnL7c5HIwxp8UKQw1xYZcm/HFAF+am7uBfX29wO44xphqzwlCDjO7bnsGJLZj42Vq+WL3T7TjGmGrKCkMNIiI8dc2ZJLSI4e63U1i/86DbkYwx1ZAVhhomIjSYKTf1IiI0mNtmeMnKtjkcjDFlY4WhBmoeU4eXbujJtv1HGDNzKfkFNmyGMSZwVhhqKE9cQx4bksC363fz1Pw1bscxxlQjIW4HMBXnurPasHr7AaZ+u4kuzepxTa9WbkcyxlQDdsZQw/35iq6c074RD3ywgpSt+92OY4ypBqww1HChwUFMvr4nTaLDGf26l10HckrvZIyp1aww1AINo8J45WYPB3PyGfV6Mjl5NmyGMebkrDDUEl2a1eOZa3uQsnU/f/5wpQ2bYYw5KSsMtciAhObc1T+e2ckZTFu42e04xpgqKqDCICIDRGStiKSLyPgSthsqIurM94yIhIrIdBFZISJpIvKA37b3iMgqEVkpIjNFJMJpbycii0VkvYi87UwnasrJXf3juaxbUx6fl8Z363e7HccYUwWVWhhEJBjfFJ0Dga7ACBHpWsx20cBYYLFf8zAgXFW7A72A0SISJyItnW09qpoABOObSxrgKWCSqsYD+4BbT3XnzC8FBQlPX5tIx9i63PHWUn7cc9jtSMaYKiaQM4beQLqqblTVXGAWMLiY7R4FJgD+H3tRIEpEQoA6QC5wwFkXAtRx1kUC20VEgIvwzQcNMB0YUrZdMqWpGx7C1Js8iMBtM7wcOprvdiRjTBUSSGFoCWz1W85w2k4QkSSgtarOLdJ3NnAY2AFsASaq6l5V3QZMdNp2AFmq+hnQCNivqsdfqX7xXH7POUpEvCLizczMDGA3jL82jSKZPLInGzIPc8/bKRw7ZjejjTE+gRQGKabtxKuIiAQBk4BxxWzXGygAWgDtgHEi0l5EGuA762jnrIsSkRtKe65CjapTVNWjqp7Y2NgAdsMUdV7Hxvz512fw+eqdPPvFOrfjGGOqiECGxMgAWvsttwK2+y1HAwnA174rQTQD5ojIIGAkMF9V84BdIrIQ8OB7sd+kqpkAIvI+cC7wJlBfREKcs4aiz2XK2S3nxpG24wDP/zedLs3rcXn35m5HMsa4LJAzhiVAvPNpoTB8N4nnHF+pqlmq2lhV41Q1DlgEDFJVL75LRReJTxTQB1jjtPcRkUjnvkJ/IE19H67/Chjq/PmbgY/KZU9NsUSER4ck0LNNfca9s5zV2w+U3skYU6OVWhicd+5jgE+BNOAdVV0lIo84ZwUlmQzUBVbiKzDTVDVVVRfju/+wFFjh5Jji9PkjcK+IpOO75/Bq2XfLlEV4SDAv3diLmDqh3DbDy97DuW5HMsa4SGrCN2A9Ho96vV63Y1R7qRn7GfbS9yS1qc/rt55NaLB9/9GYmkxEklXVU7Td/s83J5zZqj5PXXMmizbu5dG5q92OY4xxic3HYAoZktSStB0HePmbjZzRvB4jerdxO5IxppLZGYP5hT8M6EK/TrH89aOVLNm81+04xphKZoXB/EJwkPD8iCRaNYjk9jeS2bb/iNuRjDGVyAqDKVZMnVCm3uThaN4xRr/u5UiuzeFgTG1hhcGcVMcmdXluRCKrth/gD++l2hwOxtQSVhhMiS7q0pT7L+vMx8u389KCjW7HMcZUAisMplS39+vAlT1aMOHTNfx3zU634xhjKpgVBlMqEWHCNWfSrUU97pqZwpY92W5HMsZUICsMJiB1woJ56YZeIDB21jLyCo65HckYU0GsMJiAtWoQyVPXnEnK1v08/ZkN021MTWWFwZTJ5d2bM6J3G15asIFv19sEScbURFYYTJn99YqudGpal3veXs7uQ0fdjmOMKWdWGEyZ1QkL5oURPTmYk8e4d5bbtKDG1DBWGMwp6dwsmr9c0ZUF6zJ59btNbscxxpQjKwzmlF1/dhsu69aUCZ+uITVjv9txjDHlJKDCICIDRGStiKSLyPgSthsqIioiHmc5VESmi8gKEUkTkQec9s4ikuL3c0BE7nbWPSQi2/zWXV4eO2rKn4jw1DVnEls3nDtnLuNgTp7bkYwx5aDUwiAiwfim6BwIdAVGiEjXYraLBsYCi/2ahwHhqtod6AWMFpE4VV2rqomqmui0ZwMf+PWbdHy9qs471Z0zFa9+ZBjPjUhi695s/vLhShtPyZgaIJAzht5AuqpuVNVcYBYwuJjtHgUmADl+bQpEiUgIUAfIBYrONt8f2KCqP5Y1vKkazopryN0Xd+LDlO28v3Sb23GMMacpkMLQEtjqt5zhtJ0gIklAa1WdW6TvbOAwsAPYAkxU1aIzvwwHZhZpGyMiqSLymog0KC6UiIwSEa+IeDMz7fP0brvjwo70ad+Qv3y0ko2Zh9yOY4w5DYEUBimm7cT1AhEJAiYB44rZrjdQALQA2gHjRKS9X98wYBDwrl+fF4EOQCK+gvJ0caFUdYqqelTVExsbG8BumIoUHCQ8e10SYSFB3DlzGUfzbf4GY6qrQApDBtDab7kVsN1vORpIAL4Wkc1AH2COcwN6JDBfVfNUdRewEPD49R0ILFXVE0N2qupOVS1Q1WPAVHzFxVQDzWIi+MfQHqzafoCnPlnrdhxjzCkKpDAsAeJFpJ3zDn84MOf4SlXNUtXGqhqnqnHAImCQqnrxXT66SHyi8BWNNX5/ewRFLiOJSHO/xauAlaewX8Yll3Rtyi3nxvHawk02RLcx1VSphUFV84ExwKdAGvCOqq4SkUdEZFAp3ScDdfG9uC8BpqlqKoCIRAKXAO8X6TPB+XhrKnAhcE9Zdsi4b/zALpzRvB73vZvKzgM5pXcwxlQpUhM+XujxeNTr9bodw/jZkHmIK57/jsTW9Xnjd2cTHFTcrSpjjJtEJFlVPUXb7ZvPpkJ0iK3Lw4O78f3GPby0YIPbcYwxZWCFwVSYYb1aMahHC575fB3JPxb9lLIxpqqywmAqjIjw2FUJtKgfwdiZKWQdsSEzjKkOrDCYClUvIpQXRvRk54EcHng/1YbMMKYasMJgKlxi6/rcd1ln5q34iZk/bC29gzHGVVYYTKUYdUF7LohvzMMfr2LdzoNuxzHGlMAKg6kUQUHC09f2IDoihDvfWkZOng2ZYUxVZYXBVJom0RE8fW0ia3ce5LH/rHY7jjHmJKwwmErVr1Mso/q2541FW5i/cofbcYwxxbDCYCrdfZd2pkerGP4wO5Vt+4+4HccYU4QVBlPpwkKCeH5EEscU7pq5jPyCY25HMsb4scJgXNG2URSPX5WA98d9PP/lerfjGGP8WGEwrhmc2JKhvVrxwlfpfL9hj9txjDEOKwzGVQ8P6ka7RlHc83YK+w7nuh3HGIMVBuOyqPAQnh+RxN7Dudw/e7kNmWFMFWCFwbguoWUM4wd24Yu0XUz/32a34xhT6wVUGERkgIisFZF0ERlfwnZDRUSd+Z4RkVARme7MyJYmIg847Z1FJMXv54CI3O2saygin4vIeud3g/LYUVO1/ea8OPp3acLf561h1fYst+MYU6uVWhhEJBjfFJ0Dga7ACBHpWsx20cBYYLFf8zAgXFW7A72A0SISp6prVTVRVROd9mzgA6fPeOBLVY0HvnSWTQ0nIvxjWA8aRIVy58xlZOfmux3JmForkDOG3kC6qm5U1VxgFjC4mO0eBSYA/pP8KhAlIiFAHSAXOFCkX39gg6r+6CwPBqY7j6cDQwLZEVP9NYwKY9J1iWzafZiH5qxyO44xtVYghaEl4D9WcobTdoKIJAGtVXVukb6zgcPADmALMFFVi07lNRyY6bfcVFV3ADi/mxQXSkRGiYhXRLyZmZkB7IapDs7t0JgxF3bkHW8Gc5ZvdzuOMbVSIIWhuFncT3x0RESCgEnAuGK26w0UAC2AdsA4EWnv1zcMGAS8W4bMvgCqU1TVo6qe2NjYsnY3Vdhd/ePp1bYBf3p/BVv2ZLsdx5haJ5DCkAG09ltuBfi/lYsGEoCvRWQz0AeY49yAHgnMV9U8Vd0FLAQ8fn0HAktVdadf204RaQ7g/N5Vtl0y1V1IcBDPDU8kSODOWcvIsyEzjKlUgRSGJUC8iLRz3uEPB+YcX6mqWaraWFXjVDUOWAQMUlUvvstHF4lPFL6iscbvb4+g8GUknL99s/P4ZuCjU9gvU821ahDJU9ecyfKt+5n42Vq34xhTq5RaGFQ1HxgDfAqkAe+o6ioReUREBpXSfTJQF1iJr8BMU9VUABGJBC4B3i/S50ngEhFZ76x/sgz7Y2qQgd2bM/LsNry8YCPfrLP7SMZUFqkJ3zT1eDzq9XrdjmEqQE5eAYP++R17D+fxyV0XEBsd7nYkY2oMEUlWVU/Rdvvms6nSIkKDeWFETw7m5DHu3eUcO1b938gYU9VZYTBVXudm0fzliq58sy6TV77b6HYcY2o8KwymWrj+7DYMTGjGhPlrWb51v9txjKnRrDCYakFEePLqM2laL4I7Zy7jYE6e25GMqbGsMJhqIyYylOeGJ7Jt/xH+/OFKG6LbmApihcFUK564htzdP56PUrbz3tJtbscxpkaywmCqnf+7sCN92jfkrx+tZGPmIbfjGFPjWGEw1U5wkPDsdUmEhwRx58xlHM0vcDuSMTWKFQZTLTWLieAfQ3uwavsBnvxkTekdjKlhVJUDOXnk5pf/WGIh5f4XjakkF3dtyi3nxjFt4WbO79iY/mc0dTuSMWVWcEw5mJPH/uw89mXnsv9IHlnZeex3Hu/PziPryM/LWdl5vt9H8ig4prx+a28uiC/fEaatMJhq7YHLu/DDpr3c9+5yPrmrL81iItyOZGqp/IJjvhdw58V8f3au7/eRPLL8XuSLLh/IyaOkD9hFR4RQPzKU+nXCqB8ZSsv6dQotxzWKKvd9scJgqrXwkGBeGJnElS98xz1vp/DG784mOKi4KUSMCczR/AKy/N6ZH38n71v2f7H/eTkrO4+DR08+Ha0IxNQJpX6dUGIiw6gfGUZc46ifl+uE+l7sI0Opf2I5jHoRIYQEV/4VfysMptrrEFuXhwd14/7Zqbz4dTpjLop3O5KpAnLyCpwXcefFPDuPrCO57PN7fLzd/118du7JP8wQHCTOi7nvRb5JdASdmkQ7y2EnXtxjnBf2Bk57dEQIQdXoDYsVBlMjDO3Viu/SdzPpi/Wc06ERvdo2dDuSKQeqSnZugfPOPbfQu/ifX/CLfxd/tISbsqHBQkwd54XbuTzTrUW9E+/cC72Ld17wYyJDiQ4PQaT6vMCfKisMpkYQER4bksCyLfsZOzOFeWMvICYy1O1Ypgx+2LSX177bxO5DR/1uuuaSV3DyC/BhIUEn3pXHRIYS1ziS+nXqn3ghP/Eu/vi7fOcFPzIsuFa8wJ8qKwymxoiOCOX5EUkMffF/jH8/lX9d39P+568GsrLzeOKTNGYt2UpsdDgdY+sS36RukevtocT4Xao5/oIfERrsdvwaKaDCICIDgOeAYOAVVS12VjURGQq8C5ylql4RCQVeAXo6zzVDVZ9wtq3vrEsAFPitqn4vIg8BtwHHp+z6k6rOO8X9M7VMYuv63H9ZZ574ZA1v/bCF689u63YkcxKqypzl23l07mr2Zecxum977ro4nsgwe7/qtlL/C4hIML4pOi8BMoAlIjJHVVcX2S4aGAss9mseBoSrandnKs/VIjJTVTfjKzTzVXWoM5d0pF+/Sao68XR2zNRet13QnoUb9vDIx6s5K64hnZpGux3JFLF1bzYPfriSb9Zl0qNVDNN/25tuLWLcjmUcgXwOqjeQrqobVTUXmAUMLma7R4EJQI5fmwJRIhIC1AFygQMiUg/oC7wKoKq5qmqD7JtyERQkPD2sB9ERoYx5ayk5eTZkRlWRV3CMlxZs4JJJC0jevJeHruzK+/93nhWFKiaQwtAS2Oq3nOG0nSAiSUBrVZ1bpO9s4DCwA9gCTFTVvUB7fJeKponIMhF5RUT8v6UxRkRSReQ1EWlQXCgRGSUiXhHxZmbaRPGmsNjocJ65tgfrdh7i0bmrS+9gKlzK1v1c+cJ3PPnJGvrGx/LFuH7ccl47+95JFRRIYSjuv9qJjwmISBAwCRhXzHa9gQKgBdAOGCci7fFdwuoJvKiqSfiKx3inz4tAByARX0F5urhQqjpFVT2q6omNLd+vg5uaoW+nWEb3bc+bi7fwyYodbseptQ7m5PG3j1Zy1b8Wsj87j5du6MWUmzw0j6njdjRzEoHc5ckAWvsttwK2+y1H47uB/LXzCZBmwBwRGQSMxHcfIQ/YJSILAQ/wDZChqsfvR8zGKQyquvP4HxaRqUDRsxBjAjbu0s4s2riHP76XSvdWMbRqEFl6J1Nu5q/8iYfmrGLnwRxu6tOW+y7rTHSEfYy4qgvkjGEJEC8i7ZybxMOBOcdXqgNFhBgAABFQSURBVGqWqjZW1ThVjQMWAYNU1Yvv8tFF4hMF9AHWqOpPwFYR6ez8mf7AagARae733FcBK09vF01tFhYSxAsjenJM4e5ZKeQXlP9IlOaXdmQd4bYZXn7/RjL1I0N5//ZzeXhwghWFaqLUMwZVzReRMcCn+D6u+pqqrhKRRwCvqs4poftkYBq+F3cBpqlqqrPuTuBNp9hsBH7jtE8QkUR8l6s2A6PLvlvG/KxNo0gevyqBu2al8PyX67n30s6ldzKnpOCYMuP7zUz8dC0Fqowf2IVbz29HqAvj/ZhTJzVh3lyPx6Ner9ftGKaKu//d5cxemsFbv+vDOR0auR2nxlm1PYs/vb+C5RlZ9O0Uy+NDEmjd0C7dVWUikqyqnqLtVsZNrfHw4G60axzF3W8vY+/hXLfj1BjZufn8fV4ag/65kG37j/Dc8ESm/+YsKwrVmBUGU2tEhoXwwogk9h3O4w+zl1MTzpbd9tXaXVw66RumfLORYb1a8cW9/Ric2NKGIqnmrDCYWqVbixgeuLwLX6Tt4t//2+x2nGpr18Ecxry1lN9MW0JEaDDvjD6HJ685k/qRYW5HM+XABiUxtc4t58axMH03T8xbw1lxDUload+6DdSxY8qsJVt58pM0cvKOce8lnRjdrz3hITaYXU1iZwym1hERJgztQYOoUMbOXMbhEmbeMj9bv/Mg1778PX/6YAVdW9Rj/t0XMLZ/vBWFGsgKg6mVGkaF8ex1SWzac5iH5qxyO06VlpNXwNOfreXy578lPfMQ/xh6JjNv60P72LpuRzMVxC4lmVrrnA6NuPPCjjz/33TOj2/M4MSWpXeqZf6XvpsHP1zJpt2HuTqpJQ/++gwa1Q13O5apYFYYTK02tn88/9uwhwc/WEli6/q0bRRVeqdaYO/hXB7/TxrvLc2gbaNI3rj1bM6Pb+x2LFNJ7FKSqdVCgoN4dngiQQJjZy4jt4R5gmsDVeW95Az6P/01H6Vs444LO/Dp3X2tKNQyVhhMrdeqQSQThp7J8owsnv5srdtxXLNp92FueHUx495dTrvGUfxn7AXcf1kXmz6zFrJLScYAAxKac/3ZbXj5m42c27Ex/TrVnqHcc/OPMeWbDTz/33TCg4N4bEgCI3u3IcjmSai1rDAY4/jLFV3xbt7HuHdS+OSuvsRG1/ybrN7Ne/nTBytYt/MQv+7enL9d2ZUm9SLcjmVcZpeSjHFEhAbzwsgkDubkc+87KRw7VnOHzMg6ksefPljB0Je+5/DRAl692cPk63taUTCAFQZjCunUNJq/XtmVb9fvZuq3G92OU+5Ulbmp27n4mQXM+mELt57fjs/u6Uv/M5q6Hc1UIXYpyZgiRvZuw3frd/OPT9dydvtGJLau73akcpGxL5u/fLiSr9ZmktCyHq/dfBbdW9lwIOaX7IzBmCJEhCevPpOm9SIYO3MZB3Py3I50WvILjjH1m41c8sw3LN60l79c0ZUP/+88KwrmpAIqDCIyQETWiki6iIwvYbuhIqIi4nGWQ0VkuoisEJE0EXnAb9v6IjJbRNY4685x2huKyOcist753eB0d9KYsoqJDOX5EYls23+EBz9YWW2H6E7N2M/gyQt5fF4a53VsxOf39uPW89sRYjOqmRKU+q9DRILxTdE5EOgKjBCRrsVsFw2MBRb7NQ8DwlW1O9ALGC0icc6654D5qtoF6AGkOe3jgS9VNR740lk2ptL1atuQey6OZ87y7cxOznA7TpkcOprPwx+vYsjkhWQePMqL1/dk6k0eWtav43Y0Uw0E8rahN5CuqhtVNReYBQwuZrtHgQlAjl+bAlEiEgLUAXKBAyJSD+gLvAqgqrmqut/pMxiY7jyeDgwp2y4ZU35u/1VHzmnfiL9+tIoNmYfcjhOQz1fv5JJnFvDv/23m+rPb8sW4fgzs3twmzzEBC6QwtAS2+i1nOG0niEgS0FpV5xbpOxs4DOwAtgATVXUv0B7IBKaJyDIReUVEjg9S01RVdwA4v5sUF0pERomIV0S8mZmZAeyGMWUXHCRMui6RiNAg7nxrGUfzC9yOdFI/ZeXw+9eTuW2Gl3oRocz+/bk8OiSBehGhbkcz1UwghaG4txknLriKSBAwCRhXzHa9gQKgBdAOGCci7fF9Gqon8KKqJuErHmW6ZKSqU1TVo6qe2Nja8y1VU/maxUQwcVgPVu84wBPz1rgd5xcKjikzvt/Mxc8s4Ku1u/jDgM7MHXs+vdra7TlzagL5uGoG0NpvuRWw3W85GkgAvnZOVZsBc0RkEDAS332EPGCXiCwEPMA3QIaqHr8fMZufC8NOEWmuqjtEpDmw69R2zZjy0/+MpvzmvDimLdzM+R0bc3HXqvG5/7QdB3jg/RWkbN3PBfGNeWxIgo0Qa05bIGcMS4B4EWknImHAcGDO8ZWqmqWqjVU1TlXjgEXAIFX14rt8dJH4RAF9gDWq+hOwVUQ6O3+mP7DaeTwHuNl5fDPw0entojHlY/zALnRrUY/7Zy/np6yc0jtUoCO5BTz5yRqufOE7tu7N5tnrEpnx295WFEy5KLUwqGo+MAb4FN8nh95R1VUi8ohzVlCSyUBdYCW+AjNNVVOddXcCb4pIKpAI/N1pfxK4RETWA5c4y8a4LjwkmBdGJHE0/xh3v72MApeGzPhmXSaXPruAlxZs4OqeLfni3n4MSWppN5dNuZHq+vlsfx6PR71er9sxTC0xOzmD+95dzrhLOnFn//hKe97dh47y6NzVfJSynfaxUfz9qu70ad+o0p7f1DwikqyqnqLtNiSGMWV0Tc+WfLs+k2e/XM85HRrhiWtYoc+nqrzj3crf563hSG4Bd/WP5/8u7EB4iM2TYCqGff3RmDISER4bkkCrBnW4a1YKWdkVN2RG+q5DXDdlEX98bwWdm0Uz764LuOeSTlYUTIWywmDMKYiOCOX54UnsPJDDH99LLfchM47mFzDp83Vc/ty3rP3pIE9d051Zt/WhY5O65fo8xhTHCoMxp6hH6/r8YUBn5q/6iTcXbym3v7to4x4GPvctz325noHdm/HluH5cd5bNqGYqj91jMOY0/O789nyXvodH567mrLiGdG4Wfcp/a392Ln+fl8Y73gxaN6zD9N/2rlVTjJqqw84YjDkNQUHC08N6EB0Ryp0zl3Ikt+xDZqgqHyzLoP/TC3hv6TZ+368Dn93dz4qCcY0VBmNOU2x0OM9c24N1Ow/x6H9Wl97Bz497DnPTaz9wz9vLad0wkrl3ns/4gV2oE2Y3l4177FKSMeWgb6dYRvdrz8sLNnJBx8YM7N68xO3zCo4x9duNPPfFekKDg3hkcDeuP7stwXYfwVQBVhiMKSf3XdqZRRv38sf3UuneKoZWDSKL3W7pln386f0VrPnpIAO6NeOhQd1oFhNRyWmNOTm7lGRMOQkNDuKF4Umowl2zUsgvOFZo/YGcPP7y4UquefF/ZB3JY+pNHl66sZcVBVPlWGEwphy1aRTJ41d3J/nHfTz35XrAd3P5kxU7uPjpBby5+EduOTeOz+/txyVVZIRWY4qyS0nGlLNBPVrw3fpM/vlVOu1jo/hP6g6+SNtF1+b1eOVmD2e2qu92RGNKZIXBmArw0KBueH/cxz1vL6dOaDAPXn4GvzkvjpBgO0k3VZ8VBmMqQGRYCFNu7MUbi7Zw6/ntaN2w+BvRxlRFVhiMqSAdm0Tz0KBubscwpszsvNYYY0whARUGERkgImtFJF1Expew3VARURHxOMuhIjJdRFaISJqIPOC37WanPUVEvH7tD4nINqc9RUQuP50dNMYYUzalXkoSkWB8U3ReAmQAS0RkjqquLrJdNDAWWOzXPAwIV9XuIhIJrBaRmaq62Vl/oaruLuZpJ6nqxLLvjjHGmNMVyBlDbyBdVTeqai4wCxhczHaPAhMA/1nSFYgSkRCgDpALHDi9yMYYYypSIIWhJbDVbznDaTtBRJKA1qo6t0jf2cBhYAewBZioqnuddQp8JiLJIjKqSL8xIpIqIq+JSIPiQonIKBHxiog3MzMzgN0wxhgTiEAKQ3Gjep2YrkpEgoBJwLhitusNFAAtgHbAOBFp76w7T1V7AgOBO0Skr9P+ItABSMRXUJ4uLpSqTlFVj6p6YmNteGJjjCkvgRSGDKC133IrYLvfcjSQAHwtIpuBPsAc5wb0SGC+quap6i5gIeABUNXtzu9dwAf4igiqulNVC1T1GDD1eLsxxpjKEUhhWALEi0g7EQkDhgNzjq9U1SxVbayqcaoaBywCBqmqF9/lo4vEJwpf0VgjIlHOzWqc9kuBlc6y/3jFVx1vN8YYUzlK/VSSquaLyBjgUyAYeE1VV4nII4BXVeeU0H0yMA3fi7sA01Q11bmc9IGIHM/wlqrOd/pMEJFEfJerNgOjS8uYnJy8W0R+LG27k2gMFPfJKLdZrrKxXGVjucqmquaC08vWtrhGUdXi2msNEfGqqsftHEVZrrKxXGVjucqmquaCislm33w2xhhTiBUGY4wxhVhhgCluBzgJy1U2lqtsLFfZVNVcUAHZav09BmOMMYXZGYMxxphCrDAYY4wppEYXhtKGCxeRW0Qk02+I79/5rbtZRNY7PzdXoVwFfu0lfYek3HM521wrIqtFZJWIvOXX7trxKiWXa8dLRCb5Pfc6Ednvt87Nf18l5XLzeLURka9EZJkzVtrlfusecPqtFZHLqkIuEYkTkSN+x+ulSs7VVkS+dDJ9LSKt/Nad3r8vVa2RP/i+jLcBaA+EAcuBrkW2uQX4ZzF9GwIbnd8NnMcN3M7lrDvk4vGKB5YdPxZAkypyvIrN5fbxKrL9nfi+HOr68TpZLrePF76bqLc7j7sCm/0eLwfC8Y25tgEIrgK54oCVLh6vd4GbnccXAa+X17+vmnzGEOhw4cW5DPhcVfeq6j7gc2BAFchVkQLJdRsw2TkmqG+cK3D/eJ0sV0Uq63/HEcBM57Hbx+tkuSpSILkUqOc8juHnMdkGA7NU9aiqbgLSKb8x1E4nV0UKJFdX4Evn8Vd+60/731dNLgylDhfuuMY5FZstIscHCwy0b2XnAogQ33Dji0RkSDllCjRXJ6CTiCx0nn9AGfq6kQvcPV6A75Qf3zvd/5a1byXnAneP10PADSKSAczDdzYTaF83cgG0cy4xLRCRC8opU6C5lgPXOI+vAqJFpFGAfUtUkwtDicOFOz4G4lT1TOALYHoZ+rqRC6CN+r7+PhJ4VkQ6VGKuEHyXbX6F753mKyJSP8C+buQCd4/XccOB2apacAp9y+p0coG7x2sE8G9VbQVcDrwuvmH93T5eJ8u1A9/xSgLuBd4SkXqUj0By3Qf0E5FlQD9gG5AfYN8S1eTCUNpw4ajqHlU96ixOBXoF2telXOjPw5VvBL4Gkiorl7PNR+obRn0TsBbfC7Krx6uEXG4fr+OGU/hyjdvH62S53D5etwLvOM//PRCBb4A4t49XsbmcS1t7nPZkfPcEOlVWLlXdrqpXO4XpQactK8B9KllF3DipCj/43kVuxHeqfPzmTbci2zT3e3wVsEh/vnmzCd+NmwbO44ZVIFcDfHNog+9/mPWUcGOxAnINAKb7Pf9WoFEVOF4ny+Xq8XK264xvlGDxa3P1eJWQy+1/X58AtziPz8D3YiZANwrffN5I+d18Pp1cscdz4LtJvK2S/903BoKcx48Dj5TXv6/T3oGq/IPvtG8dvkr+oNP2CL75IgCeAFY5B/0roItf39/iu8mVDvymKuQCzgVWOO0rgFsrOZcAzwCrnecfXkWOV7G53D5ezvJDwJPF9HXteJ0sl9vHC9/N1IXO86cAl/r1fdDptxYYWBVy4bu+f/z/06XAlZWcayi+4r0OeAWnqJfHvy8bEsMYY0whNfkegzHGmFNghcEYY0whVhiMMcYUYoXBGGNMIVYYjDHGFGKFwRhjTCFWGIwxxhTy/4JQ9NqiaWN/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "x_axis = subsamples_s\n",
    "plt.plot(x_axis,-test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done  12 out of  15 | elapsed: 26.2min remaining:  6.6min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4854507775458169\n",
      "{'colsample_bytree': 0.5}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  15 out of  15 | elapsed: 32.5min finished\n"
     ]
    }
   ],
   "source": [
    "# sub_feature/feature_fraction/colsample_bytree\n",
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.1,\n",
    "         'max_bin':127,\n",
    "         'bagging_freq':1,\n",
    "         'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75,\n",
    "         'min_child_samples':40,\n",
    "          'subsample':0.7\n",
    "         }\n",
    "colsample_bytree_s = [i/10.0 for i in range(5,10)]\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "tuned_parameters = dict(colsample_bytree=colsample_bytree_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='neg_log_loss',verbose=5,refit=False)\n",
    "grid_search.fit(x_train,y_train)\n",
    "\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3xV9Znv8c+Ty86NXEm4JiGAeEHuhEDbabVaW+woaEXlZr3gaOfUOm3tdLQ9p9Pq6Uzr2GHaGU6nVkAqQVSqI/VltdVqbZ0hIdxBQCKQnXAN5ALkupM854+9SHdiIDsh2Wtn53m/Xnll77V+v72ftQj55rd+a68lqooxxhhzXpTbBRhjjAkvFgzGGGM6sGAwxhjTgQWDMcaYDiwYjDHGdBDjdgF9ITMzU/Py8twuwxhjBpQtW7acUtWszssjIhjy8vIoKSlxuwxjjBlQRKSsq+V2KMkYY0wHFgzGGGM6sGAwxhjTgQWDMcaYDiwYjDHGdGDBYIwxpgMLBmOMMR0EFQwiMldE9otIqYg8epF2C0RERSTfeR4rImtEZJeI7BWRxwLafkNE9ojIbhF5XkTineXPisghEdnufE271I00xoS/huZWXiwpp9HX6nYpg163wSAi0cAK4EZgIrBIRCZ20S4ZeBgoClh8OxCnqpOBmcCDIpInIqOdtvmqOgmIBhYG9Pt7VZ3mfG3v5bYZYwaQVe8f4tsbdvK157fR0trmdjmDWjAjhgKgVFUPqmozsB6Y30W7J4AngcaAZQokiUgMkAA0A2ecdTFAgrMuETjau00wxgx0rW3KuiIvmUPi+P0HJ/iHX++irc1uIuaWYIJhNFAe8LzCWdZORKYDOar6Wqe+G4A64BjgBZ5S1SpVPQI85Sw7BtSq6u8C+v1QRHaKyHIRieuqKBF5QERKRKSksrIyiM0wxoSr9z6s5EhNAz+YdzVf/9wEfr21gh++vhe7w6Q7ggkG6WJZ+7+WiEQBy4FHumhXALQCo4CxwCMiMk5E0vGPOsY665JEZKnT5zHgSmAWkAH8Q1dFqerTqpqvqvlZWR+7BpQxZgBZu6mMrOQ4Pn/1cP7u+gnc88k8Vv75ECveKXW7tEEpmIvoVQA5Ac+z6XjYJxmYBLwrIgAjgI0iMg9YDLyhqj7gpIi8D+TjD5ZDqloJICIvA58E1qrqMed1m0RkNfCt3m6cMSb8VVTX84f9J/nqtZcRG+3/W/V7N02ktsHHU7/7kNRED3fNGeNylYNLMCOGzcAEERkrIh78k8Qbz69U1VpVzVTVPFXNAzYB81S1BP+houvELwmYA+xzls8RkUTxp8n1wF4AERnpfBfgFmB3H22rMSYMvbDZf6R6YcFf/v6MihKeXDCFz101jO+9upuNO2wKMpS6DQZVbQEeAt7E/8v7RVXdIyKPO6OCi1kBDMH/y30zsFpVd6pqEf75h63ALqeOp50+hSKyy1meCfzfnm+WMWYg8LW2sX5zOZ+9YhjZ6Ykd1sVGR/Efi2cwKy+Db76wnXf3n3SpysFHImFyJz8/X+1+DMYMPK/vOsb/KtzKqnvyue7K4V22OdPoY9HTm/io8hxrl80mPy8jxFVGLhHZoqr5nZfbJ5+NMa4pLCpjdFoC11w+7IJtUuJjWXNfASNTE7jv2c3sPXbmgm1N37BgMMa44mDlOd4vPc2ighyio7o6+fEvMofE8dyyAhI9Mdy1spiy03UhqnJwsmAwxrji+WIvMVHCHbNyum8MZKcnsvb+Alrb2li6sogTZxq772R6xYLBGBNyjb5WXtpSwReuHsGw5Pig+102LJln7y2g6lwzX15ZTE19cz9WOXhZMBhjQu71XceoqfexZHZuj/tOzUnj6S/nc+hUHfc9u5n65pZ+qHBws2AwxoRcYZGXcZlJfGL80F71/9Rlmfxs0XS2l9fw4HNbaGqxK7L2JQsGY0xI7T12hi1l1SyenYtztYRemTtpBD/60hT+dOAU33xhB6120b0+E8wlMYwxps8UFpXhiYliwczsS36tO2blUNPQzD+9vo+UhFj+6dZJlxQ2xs+CwRgTMueaWnhl6xFumjKStERPn7zmA58ZT029j//37kekJ8by7blX9snrDmYWDMaYkNm4/Sh1za0smd23F8X7+y9cQU3D+XDw8DefGdenrz/YWDAYY0JCVVm7qYyrRqYwIzetT19bRHhi/iRqG3z88PW9pCbGckd+cJ+PMB9nk8/GmJDYXl7DB8fOsOQSJ50vJDpKWH7HND49IZNHf72TN3Yf7/P3GCwsGIwxIVFY5CXJE80t00d337iXPDFR/OKumUzNSePh57fxfumpfnuvSGbBYIzpd7X1Pn6z4yjzp49mSFz/HsFO9MSw+p5ZjM1M4oFflbCjvKZf3y8SWTAYY/rdhq0VNLW0sbSPJ50vJC3Rw6+WFZAxxMM9q4spPXk2JO8bKSwYjDH9SlUpLCpjem4aE0elhOx9h6fEs3bZbGKio1j6TDEV1fUhe++BzoLBGNOvNh2s4mBlXZ+fohqMMUOT+NV9BdQ3t3DXymJOnWsKeQ0DkQWDMaZfFRaVkZoQy01TRrry/leNTGHVPbM4VtvA3auKOdPoc6WOgcSCwRjTbyrPNvHmnuMsmJlNfGy0a3Xk52Xwn0tnsv/4We5fU0Kjzy66dzEWDMaYfvNiSTm+VmVxLy6v3deuvWIY/3rnNDYfruKhdVvxtba5XVLYsmAwxvSL1jbl+WIvnxg3lPFZQ9wuB4B5U0fx+PxJvLX3JN/esJM2uyJrl+ySGMaYfvHegUoqqht47Mar3C6lg7vmjKG2vpmnfvchqQmx/OPNE+2KrJ1YMBhj+kXhpjIyh8Rxw8ThbpfyMV/97GVU1/tY+edDpCd6+LvPTXC7pLAS1KEkEZkrIvtFpFREHr1IuwUioiKS7zyPFZE1IrJLRPaKyGMBbb8hIntEZLeIPC8i8c7ysSJSJCIHROQFEemba/MaY0LmSE0Df9h3kjtnZeOJCb8j1iLCd794FbfNyGb5Wx+y5r8Pu11SWOn2X0xEooEVwI3ARGCRiEzsol0y8DBQFLD4diBOVScDM4EHRSRPREY7bfNVdRIQDSx0+vwYWK6qE4BqYFlvN84Y444Xir0osHCW+5POFxIVJfz4tsncMHE4/7hxD69uP+J2SWEjmCgvAEpV9aCqNgPrgfldtHsCeBJoDFimQJKIxAAJQDNwxlkXAyQ46xKBo+I/0HcdsMFpswa4pWebZIxxk6+1jfWby/nsFcPIyUh0u5yLiomO4t8XTWfOuAweeXEH7+w76XZJYSGYYBgNlAc8r3CWtROR6UCOqr7Wqe8GoA44BniBp1S1SlWPAE85y44Btar6O2AoUKOqLRd6r4D3fEBESkSkpLKyMojNMMaEwlsfnODk2SaWhMEpqsGIj43ml1/O56qRKXxl7RaKD1W5XZLrggmGrqbr28/xEpEoYDnwSBftCoBWYBQwFnhERMaJSDr+UcdYZ12SiCzt7r06LFR9WlXzVTU/KysriM0wxoRCYZGXUanxXHvFMLdLCVpyfCzP3juL0ekJLHt2M3uO1rpdkquCCYYKIPBWSNnA0YDnycAk4F0ROQzMATY6E9CLgTdU1aeqJ4H3gXzgc8AhVa1UVR/wMvBJ4BSQ5hxe6uq9jDFh7NCpOv5ceopFBblERw2sU0CHDolj7bLZJMfHcPeqYg6dqnO7JNcEEwybgQnO2UIe/JPEG8+vVNVaVc1U1TxVzQM2AfNUtQT/oaLrxC8Jf2jsc5bPEZFEZ17hemCvqirwDrDAefm7gVf7ZEuNMf1uXVEZMVHCnbMG5m01R6Ul8Nz9s2lTWPpMEcdrG7vvFIG6DQbneP9DwJvAXuBFVd0jIo+LyLxuuq8AhgC78QfMalXdqapF+OcftgK7nDqedvr8A/BNESnFP+ewsuebZYwJtUZfKy9tqeDzVw9nWEq82+X02visIay5t4DaBh93rSyiuq7Z7ZJCTvx/pA9s+fn5WlJS4nYZxgxqr2yr4Bsv7KDw/tl86rJMt8u5ZP/z0WnuXl3MxJEpFN4/m6R+vvOcG0Rki6rmd14efp88McYMSIWbvIzLTOKT44e6XUqf+MT4ofzHounsOlLLg89toall8FyR1YLBGHPJ9h0/Q0lZNYtn50bUdYc+f/UIfnzbFP5ceoqvr99O6yC56J4FgzHmkhVu8uKJieK2Gdlul9LnFszM5v/cNJHf7j7Od17eRSQcfu9O5B00M8aEVF1TC69sO8JNk0eSnhSZlzZb9ldjqalv5t//UEpaUmzYXTG2r1kwGGMuycYdRznX1MKSOaG/p3MoffOGy6mp9/GLPx4kPdHDV64Z73ZJ/caCwRjTa6rK2k1lXDkimRm5aW6X069EhB/Mu5qaBh8/+u0+0hJiWVgwMC770VM2x2CM6bUdFbXsOXqGJXPGRNSk84VERQk/uX0q11yexXde2cVvdx1zu6R+YcFgjOm1wk1lJHqiuWXaKLdLCRlPTBT/uXQm03PT+bv12/nzgVNul9TnLBiMMb1SW+/jNzuPcsv00STHx7pdTkgleKJZdfcsxmUl8cBzJWzzVrtdUp+yYDDG9Mqvt1bQ6GtjcYQeZ+9OamIsv1pWQFZyHPes3syHJ866XVKfsWAwxvSYqlJYVMa0nDQmjU51uxzXDEuOZ+2y2cTFRHHXyiLKq+rdLqlPWDAYY3qs6FAVH1XWsTTCT1ENRk5GIs8tm02jr427VhZRebbJ7ZIumQWDMabH1m4qIyU+hpumjHS7lLBwxYhkVt0zixNnmvjyqmJqG3xul3RJLBiMMT1SebaJN/ccZ8HMHOJjo90uJ2zMHJPOL+6aSenJs9y/ZjMNzQP3onsWDMaYHnlpSzm+VmXxALmncyh95vIslt85jZKyav5X4RZ8rW1ul9QrFgzGmKC1tSnrirx8YtxQLhs2xO1ywtJNU0bxw1sm887+Sr710g7aBuAVWe2SGMaYoP3xQCUV1Q08euOVbpcS1hbPzqWmoZkn39hPakIsP5h39YD6ZLgFgzEmaIWbvGQO8fD5iSPcLiXs/e0146mp9/H0ewdJS/TwzRsud7ukoFkwGGOCcrSmgT/sO8FXrhmPJ8aOQndHRHjsxiupqW/mZ28fID0xlns/NdbtsoJiwWCMCcr6zeUosGiQftK5N0SEf7p1MrUNPn7wmw9ITYjlSwPgZkYW+8aYbvla21hf7OXay7PIyUh0u5wBJSY6ip8unM4nxw/l7zfs5K0PTrhdUrcsGIwx3Xp77wlOnm1iyWz7pHNvxMdG8/SX85k0KoWvrtvKpoOn3S7poiwYjDHdKizyMio1ns9eOcztUgasIXExPHtvATkZidy/poTdR2rdLumCggoGEZkrIvtFpFREHr1IuwUioiKS7zyPFZE1IrJLRPaKyGPO8itEZHvA1xkR+bqz7vsiciRg3Rf7YkONMb1z+FQdfzpwikUFuURHDZxTLsNRepKH55YVkJoQy92rijlYec7tkrrUbTCISDSwArgRmAgsEpGJXbRLBh4GigIW3w7EqepkYCbwoIjkqep+VZ2mqtOc5fXAKwH9lp9fr6qv93bjjDGXbl2xl+go4c5ZOW6XEhFGpibw3LICAO5aWczRmgaXK/q4YEYMBUCpqh5U1WZgPTC/i3ZPAE8CjQHLFEgSkRggAWgGznTqdz3wkaqW9bR4Y0z/avS18lJJOZ+fOJxhKfFulxMxxmUNYc19BZxp8HHXyiKq6prdLqmDYIJhNFAe8LzCWdZORKYDOar6Wqe+G4A64BjgBZ5S1apObRYCz3da9pCI7BSRVSKS3lVRIvKAiJSISEllZWUQm2GM6ak3dh+nut5nl9fuB5NGp/LM3flUVDdwz+pizjW1uF1Su2CCoauDiu0X/xCRKGA58EgX7QqAVmAUMBZ4RETGBfT1APOAlwL6/BwYD0zDHyg/6aooVX1aVfNVNT8rKyuIzTDG9FRhURljM5P4xLihbpcSkWaPG8qKxTPYc/QMD/yqhEZfeFyRNZhgqAACDy5mA0cDnicDk4B3ReQwMAfY6ExALwbeUFWfqp4E3gfyA/reCGxV1fYTe1X1hKq2qmob8Ev84WKMCbF9x8+w+XA1iwtyibJJ537zuYnDeer2Kfz3R6d5+PlttITBFVmDCYbNwAQRGev8hb8Q2Hh+parWqmqmquapah6wCZinqiX4Dx9dJ35J+ENjX8BrL6LTYSQRCbzzx63A7l5slzHmEq0r8uKJiWLBzPD/pO5Ad+v0bL5/80R+98EJHnt5F6ruXpG120tiqGqLiDwEvAlEA6tUdY+IPA6UqOrGi3RfAazG/8tdgNWquhNARBKBG4AHO/V5UkSm4T9cdbiL9caYflbX1MLLW49w0+SRpCd53C5nULjnU2Oprvfx07cPkJYYy3e+eJVrV2QN6lpJzimjr3da9r0LtL024PE5/KesdtWuHvjYgUtVvSuYmowx/WfjjqOca2phyRy7LlIoff1zE6ipb+aXfzpEWqKHr372MlfqsIvoGWM6UFXWbirjyhHJzMjt8qRA009EhH+8+WpqG3z8y5v7SUuMdeUyJBYMxpgOdlbUsufoGZ6YP7BuLhMpoqKEf7l9KmcaW/jf/7Wb1IRYbpoyKrQ1hPTdjDFhr7CojERPNLdMH919Y9MvYqOjWLF4BrPGZPCNF7bzxw9D+1ktCwZjTLvaeh8bdxxl/rTRJMfHul3OoJbgieaZe/KZMCyZrzy3hS1l1SF7bwsGY0y7l7dV0OhrY8lsm3QOBynxsay5r4DhKXHcu7qYfcc7X1Gof1gwGGMA/6RzYZGXqTlpTBqd6nY5xpGVHMdzy2aT6InhrpXFeE/X9/t7WjAYYwAoPlRF6clzLLXRQtjJyUjkuWUF+FrbWLqyiJNnGrvvdAksGIwxAKwt8pISHxPyM2BMcCYMT+bZews4da6JL68qprbe12/vZcFgjOHUuSbe2H2M22Zmk+CJdrsccwHTctJ4+q58DlbWcd+azdQ3988VWS0YjDG8VFKBr1Xtns4DwF9NyORni6axzVvN367dSnNL3190z4LBmEGurU1ZV1zGnHEZXDZsiNvlmCDMnTSSf/7SZP74YSVv7T3RfYcesk8+GzPIvXegkvKqBr79hSvdLsX0wJ2zcrl6VGq/nEFmIwZjBrnCIi+ZQzx84eoRbpdieqi/Tiu2YDBmEDtW28Dbe09wR34Onhj7dWD87CfBmEFsfXE5CiwqsM8umL+wYDBmkPK1trF+s5drLs8iJyPR7XJMGLFgMGaQenvvSU6cabJTVM3HWDAYM0gVFpUxKjWe664c5nYpJsxYMBgzCB0+VcefDpxiYUEu0VF2Mx7TkQWDMYPQ88VeoqOEO2fluF2KCUMWDMYMMk0trbxYUs4NVw1neEq82+WYMGTBYMwg88bu41TX+1g6xyadTdcsGIwZZNZuKiNvaCKfHD/U7VJMmAoqGERkrojsF5FSEXn0Iu0WiIiKSL7zPFZE1ojILhHZKyKPOcuvEJHtAV9nROTrzroMEfm9iBxwvqf3xYYaY2D/8bNsPlzN4tm5RNmks7mAboNBRKKBFcCNwERgkYhM7KJdMvAwUBSw+HYgTlUnAzOBB0UkT1X3q+o0VZ3mLK8HXnH6PAq8raoTgLed58aYPrCuqAxPTBQLZtqks7mwYEYMBUCpqh5U1WZgPTC/i3ZPAE8CgfecUyBJRGKABKAZ6Hw36+uBj1S1zHk+H1jjPF4D3BLMhhhjLq6+uYWXtx7hryePJCPJ43Y5JowFEwyjgfKA5xXOsnYiMh3IUdXXOvXdANQBxwAv8JSqVnVqsxB4PuD5cFU9BuB87/LTNyLygIiUiEhJZWVlEJthzOC2cftRzja1sMTu6Wy6EUwwdHUgUttXikQBy4FHumhXALQCo4CxwCMiMi6grweYB7zUg5r9Bag+rar5qpqflZXV0+7GDDqFRV6uGJ7MzDE2bWcuLphgqAACD0hmA0cDnicDk4B3ReQwMAfY6ExALwbeUFWfqp4E3gfyA/reCGxV1cBbEJ0QkZEAzveTPdskY0xnOytq2HWklqVzchGxSWdzccEEw2ZggoiMdf7CXwhsPL9SVWtVNVNV81Q1D9gEzFPVEvyHj64TvyT8obEv4LUX0fEwEs5r3+08vht4tRfbZYwJULjJS6Inmlumj+6+sRn0ug0GVW0BHgLeBPYCL6rqHhF5XETmddN9BTAE2I0/YFar6k4AEUkEbgBe7tTnR8ANInLAWf+jHmyPMaaT2gYfr+44wvxpo0iOj3W7HDMABHXPZ1V9HXi907LvXaDttQGPz+E/ZbWrdvXAxz5ho6qn8Z+pZIzpA69sraDR12aX1zZBs08+GxPBVJXCIi9Tc9L67f7AJvJYMBgTwYoPVXHg5Dk7RdX0iAWDMRGssMhLcnwMN08Z5XYpZgCxYDAmQp0618Rvdx/jthnZJHii3S7HDCAWDMZEqA1bKvC1Kkvn2GEk0zMWDMZEoLY2ZV2Rl9ljM7hsWLLb5ZgBxoLBmAj0p9JTeKvqWWI34zG9YMFgTAQq3FTG0CQPc68e4XYpZgCyYDAmwhyrbeDtfSe5Y1YOnhj7L256zn5qjIkw64vLaVNl0SybdDa9Y8FgTARpaW1j/WYvn5mQRe7QRLfLMQOUBYMxEeTtfSc5caaJpTbpbC6BBYMxEaSwyMvI1Hg+e4XdvMr0ngWDMRGi7HQd731YycJZucRE239t03v202NMhFhX7CU6SrhzVk73jY25CAsGYyJAU0srL5VUcMNVwxmRGu92OWaAs2AwJgK8sfs4VXXNLLHrIpk+YMFgTAQo3ORlzNBEPjU+0+1STASwYDBmgPvwxFmKD1exuCCXqChxuxwTASwYjBng1hV58URHcXu+TTqbvmHBYMwAVt/cwq+3VPDFySPISPK4XY6JEBYMxgxgv9lxlLNNLXZ5bdOnLBiMGcAKi7xcMTyZ/DHpbpdiIkhQwSAic0Vkv4iUisijF2m3QERURPKd57EiskZEdonIXhF5LKBtmohsEJF9zrpPOMu/LyJHRGS78/XFS91IYyLRzooadlbUsmROLiI26Wz6Tkx3DUQkGlgB3ABUAJtFZKOqftCpXTLwMFAUsPh2IE5VJ4tIIvCBiDyvqoeBnwJvqOoCEfEAgZeCXK6qT13KhhkT6Qo3eUmIjeaW6aPdLsVEmGBGDAVAqaoeVNVmYD0wv4t2TwBPAo0ByxRIEpEYIAFoBs6ISArwGWAlgKo2q2pN7zfDmMGltsHHxh1HmT9tFCnxsW6XYyJMMMEwGigPeF7hLGsnItOBHFV9rVPfDUAdcAzwAk+pahUwDqgEVovINhF5RkSSAvo9JCI7RWSViNjBU2M6+a9tR2jwtbJktk06m74XTDB0dfBS21eKRAHLgUe6aFcAtAKjgLHAIyIyDv8hrBnAz1V1Ov7wOD938XNgPDANf6D8pMuiRB4QkRIRKamsrAxiM4yJDKpKYVEZU7NTmZyd6nY5JgIFEwwVQOAnZ7KBowHPk4FJwLsichiYA2x0JqAX459H8KnqSeB9IN95zQpVPT8fsQF/UKCqJ1S1VVXbgF/iD5ePUdWnVTVfVfOzsuza82bw2Hy4mg9PnLPRguk3wQTDZmCCiIx1JokXAhvPr1TVWlXNVNU8Vc0DNgHzVLUE/+Gj68QvCX9o7FPV40C5iFzhvMz1wAcAIjIy4L1vBXZf2iYaE1kKi8pIjo/hpqkju29sTC90e1aSqraIyEPAm0A0sEpV94jI40CJqm68SPcVwGr8v9wFWK2qO511XwMKnbA5CNzrLH9SRKbhP1x1GHiw55tlTGQ6fa6J3+46zuLZuSR6uv3va0yvBPWTpaqvA693Wva9C7S9NuDxOfynrHbVbjv+w0qdl98VTE3GDEYbtlTQ3NrGktl2eW3Tf+yTz8YMEG1tyrpiLwVjM5gwPNntckwEs2AwZoD4c+kpyk7X22jB9DsLBmMGiMKiMoYmeZg7aYTbpZgIZ8FgzABwvLaRt/ae5Pb8HOJiot0ux0Q4CwZjBoD1m720qbK4wA4jmf5nwWBMmGtpbWN9cTmfmZBF7tDE7jsYc4ksGIwJc3/Yd5LjZxpt0tmEjAWDMWFubZGXESnxXHflMLdLMYOEBYMxYcx7up73PqxkYUEOMdH239WEhv2kGRPG1hV7iY4SFs6yw0gmdCwYjAlTTS2tvFRSzueuGsaI1Hi3yzGDiAWDMWHqjd3HOV3XbJfXNiFnwWBMmCos8jJmaCJ/dVmm26WYQcaCwZgwdODEWYoPVbG4IJeoqK5uomhM/7FgMCYMFRZ58URHsWBmttulmEHIgsGYMFPf3MKvt1Zw4+QRDB0S53Y5ZhCyYDAmzLy24xhnG1ts0tm4xoLBmDBTWFTG5cOHMCsv3e1SzCBlwWBMGNlVUcuOilqWzB6DiE06G3dYMBgTRgqLykiIjebWGaPdLsUMYhYMxoSJM40+Xt1+lPnTRpESH+t2OWYQs2AwJkz817YjNPhabdLZuM6CwZgwoKoUbvIyJTuVydmpbpdjBjkLBmPCQElZNftPnLWb8ZiwEFQwiMhcEdkvIqUi8uhF2i0QERWRfOd5rIisEZFdIrJXRB4LaJsmIhtEZJ+z7hPO8gwR+b2IHHC+2zl7JuIVbiojOT6Gm6eOcrsUY7oPBhGJBlYANwITgUUiMrGLdsnAw0BRwOLbgThVnQzMBB4UkTxn3U+BN1T1SmAqsNdZ/ijwtqpOAN52nhsTsarqmnl913Fum5FNoifG7XKMCWrEUACUqupBVW0G1gPzu2j3BPAk0BiwTIEkEYkBEoBm4IyIpACfAVYCqGqzqtY4feYDa5zHa4BberZJxgwsL5WU09zaxmI7jGTCRDDBMBooD3he4SxrJyLTgRxVfa1T3w1AHXAM8AJPqWoVMA6oBFaLyDYReUZEkpw+w1X1GIDzvcsb3YrIAyJSIiIllZWVQWyGMeGnrU1ZV+ylIC+Dy4cnu12OMUBwwdDVxy+1faVIFLAceKSLdgVAKzAKGAs8IiLjgBhgBvBzVZ2OPzx6dMhIVZ9W1XxVzc/KyupJV2PCxvsfnaLsdD1L5thowYSPYIKhAsgJeJ4NHA14ngxMAsVSf98AAA3FSURBVN4VkcPAHGCjMwG9GP88gk9VTwLvA/nOa1ao6vn5iA34gwLghIiMBHC+n+zNhhkzEBRu8pKR5GHupBFul2JMu2CCYTMwQUTGiogHWAhsPL9SVWtVNVNV81Q1D9gEzFPVEvyHj64TvyT8obFPVY8D5SJyhfMy1wMfOI83Anc7j+8GXr20TTQmPB2vbeT3e09we342cTHRbpdjTLtuT4FQ1RYReQh4E4gGVqnqHhF5HChR1Y0X6b4CWA3sxn9IarWq7nTWfQ0odMLmIHCvs/xHwIsisgx/sNzei+0yJuy9sLmc1jZlcYEdRjLhRVS1+1ZhLj8/X0tKStwuw5igtbS28ekn32HC8GR+dV+B2+WYQUpEtqhqfufl9slnY1zwzv5KjtU22iedTViyYDDGBWs3lTEiJZ7rr+zybGxjXGXBYEyIeU/X896BShYW5BATbf8FTfixn0pjQuz5zV6iRFg4yw4jmfBkwWBMCDW3tPHi5nKuv3IYI1Lj3S7HmC5ZMBgTQm/sOc7pumaWzLGb8ZjwZcFgTAgVbiojNyORT1+W6XYpxlyQBYMxIVJ68ixFh6pYPDuXqKiuLkFmTHiwYDAmRAqLvMRGC7fPzHa7FGMuyoLBmBBoaG7l11squHHSSIYOiXO7HGMuyoLBmBD4zc6jnGlsYalNOpsBwO4jaC6qpbWNfcfPstVbzdayaiqqG0hLjCUjyUNGUhwZSbEdvyd6yBjiIckTjYgdRz+vsMjLhGFDmJVntzA34c+CwXRw+lwTW7017UGws6KWBl8rAMOS4xibmcSRmkZ2Hamlqq4ZX2vXF2H0xET5QyLp41/pSR6GJnlIT/QwdIj/e3pibMR+Cnj3kVp2lNfw/ZsnWliaAcGCYRA7PxrY5q1uD4Oy0/UAxEQJV49K4c5ZOcwYk86M3DRGpyV0+MWmqpxraqG6zsfpuiaq65s5fa6ZqrpmquqbqTrX7F9W10xFdT2n65o529hywXpSE2IZ2jk8AkIkY4inQ9gkDpBRSWFRGQmx0XzJJp3NAGHBMIhU1TWztazaPxrw+kcD9c3+0UDmkDhm5KaxuCCXGWPSmTw6lfjYi988RkRIjo8lOT6W3KGJQdXQ3NJGjRMW1XXOdydQqgOWl1fVs6O8hqq6Zlrauh6VxMVEtYdHh1FJFyGSkeQhLdFDdIhPEz3T6OPV7UeZN3UUKfGxIX1vY3rLgiFCtbYp+8/PDXir2eat4dCpOgCio4SJI1O4fWa2MxpIJzs9ISR/fXtiohiWEs+wlOAuB6GqnGls+UuI1AWMSOo6fpWdrqe6rpmzTV2PSkQgLSG268NbAYe1hibFtQdLgufS7qz26rYj1De32j2dzYBiwRAhquua2VZezdYy/yGhHeU11LWPBjxMz03njvwcZuSmMSU77ZJ/4YWKiJCaEEtqQix5mUlB9WlqaaW6zveX0KhvpupcE1X1PqrqmtoPfR0+Vc+Wshqq65tpvcCoJCE2+qJzIxlJgYHiITUhtv3Da6rK2k1eJo9OZUp2Wp/tE2P6mwXDANTapnx44vyZQjVs81ZzMGA0cNXIZG6bmc2MXP9oICcjNKOBcBEXE82I1OigL1LX1qacbWzpME9S3elwV5Xz+GDlOarrmttDt7MowT+ZnuQhOT6G/SfO8uPbJvfl5hnT7ywYBoCa+ma2nT9TyFvNjvJazjmHS4Ym+UcDC/L9QTAlO5VEj/2z9kRUlJCaGEtqYvBzAI2+1g4h0vmwVpUTKJ+ekMnNU0f1Y/XG9D37DRJm2tqUAyfPsdVbzRZnovhgpX80ECVw5YgUbp0+mhlj0piRm05uRuKgGg2Ei/jYaEamJjAyNcHtUozpcxYMLqut9/nnBrz+Q0LbvTXtk6fpibHMyE3nthnZTM9NY2p2Gklx9k9mjOlf9lsmhNralNLKcwGnjNZQevIc4B8NXDEihXnTRvnnBsakkzfURgPGmNCzYOhHtQ0+tpfXtAfB9vKa9g94pTmjgVucIJiSk8YQGw0YY8JAUL+JRGQu8FMgGnhGVX90gXYLgJeAWapaIiKxwDPADOe9fqWq/+y0PQycBVqBFlXNd5Z/H/gboNJ52e+o6uu92roQamtTPqo8136m0FZvNaWV51D1jwYuH57MzVOd0UBuGmMzk2w0YIwJS90Gg4hEAyuAG4AKYLOIbFTVDzq1SwYeBooCFt8OxKnqZBFJBD4QkedV9bCz/rOqeqqLt12uqk/1fHNC50yjj+3tZwrVsN1bzRlnNJCaEMuM3DTmTR3FjDHpTLXRgDFmAAnmt1UBUKqqBwFEZD0wH/igU7sngCeBbwUsUyBJRGKABKAZOHOpRYdaW5ty8FSd8wli/9lCB076RwMicPmwZP56ykimO58bGJeZZHfoMsYMWMEEw2igPOB5BTA7sIGITAdyVPU1EQkMhg34Q+QYkAh8Q1WrnHUK/E5EFPiFqj4d0O8hEfkyUAI8oqrVnYsSkQeABwByc/v2cgNnG33sKK/tcDmJ2gYfACnxMUzPTeevJ49ixpg0puak2TVwjDERJZhg6OpP3/brB4hIFLAcuKeLdgX45xBGAenAn0TkLWf08SlVPSoiw4Dfi8g+VX0P+Dn+0Yc6338C3PexAvxB8jRAfn5+19czCIKqMxoo+8spo/tPnG0fDUwYNoQbJ41wzhRKY1zmEBsNGGMiWjDBUAHkBDzPBo4GPE8GJgHvOpOpI4CNIjIPWAy8oao+4KSIvA/kAwdV9SiAqp4UkVfwh8h7qnri/AuLyC+B13q7cd352dsHWPX+IWrq/aOBZGc0MNcJgmm5Nhowxgw+wQTDZmCCiIwFjgAL8f/CB0BVa4HM889F5F3gW85ZSdcD14nIWvyHkuYA/yYiSUCUqp51Hn8eeNzpP1JVjzkvdyuw+xK38YKGp8TxhYkj2j9FPD7LRgPGGNNtMKhqi4g8BLyJ/3TVVaq6R0QeB0pUdeNFuq8AVuP/5S7AalXdKSLjgFecEUYMsE5V33D6PCki0/AfSjoMPNi7TevenbNyuXOWXQ7ZGGMCiWqvD8+Hjfz8fC0pKXG7DGOMGVBEZMv5z5AFisyb7BpjjOk1CwZjjDEdWDAYY4zpwILBGGNMBxYMxhhjOrBgMMYY04EFgzHGmA4i4nMMIlIJlPWyeybQ1aW/3WZ19YzV1TNWV8+Ea11wabWNUdWszgsjIhguhYiUdPUBD7dZXT1jdfWM1dUz4VoX9E9tdijJGGNMBxYMxhhjOrBgcO7pEIasrp6xunrG6uqZcK0L+qG2QT/HYIwxpiMbMRhjjOnAgsEYY0wHER0MIjJXRPaLSKmIPNrF+ntEpFJEtjtf9wesu1tEDjhfd4dRXa0Byy92k6Q+r8tpc4eIfCAie0RkXcBy1/ZXN3W5tr9EZHnAe38oIjUB69z8+bpYXW7ur1wReUdEtonIThH5YsC6x5x++0XkC+FQl4jkiUhDwP76zxDXNUZE3nZqeldEsgPWXdrPl6pG5Bf+u819BIwDPMAOYGKnNvcA/9FF3wzgoPM93Xmc7nZdzrpzLu6vCcC28/sCGBYm+6vLutzeX53afw3/3Q9d318Xqsvt/YV/EvVvnccTgcMBj3cAccBY53Wiw6CuPGC3i/vrJeBu5/F1wHN99fMVySOGAqBUVQ+qajOwHpgfZN8vAL9X1SpVrQZ+D8wNg7r6UzB1/Q2wwtknqOpJZ7nb++tCdfWnnv47LgKedx67vb8uVFd/CqYuBVKcx6nAUefxfGC9qjap6iGg1Hk9t+vqT8HUNRF423n8TsD6S/75iuRgGA2UBzyvcJZ1dpszFNsgIjk97BvqugDiRaRERDaJyC19VFOwdV0OXC4i7zvvP7cHfd2oC9zdX4B/yI//L90/9LRviOsCd/fX94GlIlIBvI5/NBNsXzfqAhjrHGL6o4h8uo9qCrauHcBtzuNbgWQRGRpk34uK5GCQLpZ1Pjf3N0Ceqk4B3gLW9KCvG3UB5Kr/4++LgX8TkfEhrCsG/2Gba/H/pfmMiKQF2deNusDd/XXeQmCDqrb2om9PXUpd4O7+WgQ8q6rZwBeB50QkKsi+btR1DP/+mg58E1gnIin0jWDq+hZwjYhsA64BjgAtQfa9qEgOhgog8C/tbDoNAVX1tKo2OU9/CcwMtq9LdaGqR53vB4F3gemhqstp86qq+pwh/X78v5Bd3V8Xqcvt/XXeQjoernF7f12oLrf31zLgRef9/weIx3+BOLf3V5d1OYe2TjvLt+CfE7g8VHWp6lFV/ZITTN91ltUGuU0X1x8TJ+Hwhf+vyIP4h8rnJ2+u7tRmZMDjW4FN+pfJm0P4J27SnccZYVBXOhDnPM4EDnCRicV+qGsusCbg/cuBoWGwvy5Ul6v7y2l3BXAY58Ok4fDzdZG63P75+i1wj/P4Kvy/zAS4mo6Tzwfpu8nnS6kr63wd+CeJj4T45z4TiHIe/xB4vK9+vi55A8L5C/+w70P8Sf5dZ9njwDzn8T8De5yd/g5wZUDf+/BPcpUC94ZDXcAngV3O8l3AshDXJcC/Ah84778wTPZXl3W5vb+c598HftRFX9f214Xqcnt/4Z9Mfd95/+3A5wP6ftfptx+4MRzqwn98//z/063AzSGuawH+8P4QeAYn1Pvi58suiWGMMaaDSJ5jMMYY0wsWDMYYYzqwYDDGGNOBBYMxxpgOLBiMMcZ0YMFgjDGmAwsGY4wxHfx/IO/uSJHMjfYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "x_axis = colsample_bytree_s\n",
    "plt.plot(x_axis,-test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 4 candidates, totalling 12 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   8 out of  12 | elapsed: 14.0min remaining:  7.0min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4854507775458169\n",
      "{'colsample_bytree': 0.5}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  12 out of  12 | elapsed: 21.3min finished\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.1,\n",
    "         'max_bin':127,\n",
    "         'bagging_freq':1,\n",
    "         'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75,\n",
    "         'min_child_samples':40,\n",
    "          'subsample':0.7\n",
    "         }\n",
    "colsample_bytree_s = [i/10.0 for i in range(2,6)]\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "tuned_parameters = dict(colsample_bytree=colsample_bytree_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='neg_log_loss',verbose=5,refit=False)\n",
    "grid_search.fit(x_train,y_train)\n",
    "\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD5CAYAAAAuneICAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU5fnG8e+TjSQQEpYEEMK+hFWWEHBBNhe0FbRat4KyqLVqtQpt7d5q689aFLWlroBCrdbaRbTuQFARCUHZCTvIoiQsCUsSkpD398cMNNJIBrKcmcz9ua5czDlzTuZ5M2HuzHnnPMecc4iISPiJ8LoAERHxhgJARCRMKQBERMKUAkBEJEwpAEREwpQCQEQkTEUFspGZjQIeByKB55xzD33NdlcDfwcGOueyzSwGeBpIB8qBu51zmSftMxfo6JzrVVUdzZs3d+3btw+kZBER8Vu2bNle51zyyeurDAAziwSmAxcBO4GlZjbXObf2pO0SgLuAJRVW3wLgnOttZinAW2Y20DlX7t/nW8DhQAfRvn17srOzA91cREQAM9te2fpADgFlAJucc1uccyXAy8CYSrZ7AHgYKK6wrgcwD8A5lwvk43s3gJk1Au4FfhvgGEREpAYFEgCtgR0Vlnf6151gZv2AVOfcGyftuwIYY2ZRZtYBGACk+u97AHgEKDzVg5vZrWaWbWbZeXl5AZQrIiKBCCQArJJ1J/pHmFkEMA2YXMl2M/EFRjbwGPAxUGZmfYHOzrl/VfXgzrlnnHPpzrn05OT/OYQlIiJnKJBJ4J389692gDbA7grLCUAvINPMAFoCc81stHMuG7jn+IZm9jGwERgKDDCzbf4aUsws0zk37MyHIiIipyOQAFgKdPEfwtkFXAfccPxO51wB0Pz4spllAlP8nwKKB8w5d8TMLgLK/JPHa4En/du3B97Qi7+ISN2qMgCcc2VmdifwDr6Pgc50zq0xs/uBbOfc3FPsngK8Y2bl+MJjXE0ULSIi1Weh1A46PT3d6WOgIiKnx8yWOefST14fFmcCv5T1OQs36BNEIiIV1fsAKCkr5y+fbOe2OctYtv2A1+WIiASNeh8AMVERPD8hgxaNGzBhVhY5Xx70uiQRkaBQ7wMAIDmhAXMmDSI+JopxM7L4fN8pzz0TEQkLYREAAKlN45kzKYPSY+WMnbGE3IPFVe8kIlKPhU0AAHRpkcDzEzLYe/go42ZkUVBY6nVJIiKeCasAAOibmsSzN6azde8RJjyfRWFJmdcliYh4IuwCAOC8zs154vp+LN+Rz3fnLKOkrNzrkkRE6lxYBgDAqF4teehbffhw417ueWU5x8pD54Q4EZGaENAVweqrawamUlBUyu/eXEdiXDS/u6IX/oZ2IiL1XlgHAMAtF3TkQGEJf87cTFJcND8aleZ1SSIidSLsAwDgh5d0I7+o1BcC8dHcekEnr0sSEal1CgDAzHhgTC8OFpXy4Js5JMXFcM3A1Kp3FBEJYQoAv8gI49Fr+nKwuIz7/rmSxnFRjOrVyuuyRERqTdh+CqgyMVERPDW2P31Tk7jrpeUs2rTX65JERGqNAuAk8TFRzBqfQcfkhtwyO5vlO/K9LklEpFYoACqRGB/N7IkZNG/UgPGzsti455DXJYmI1DgFwNdIaRzLXyYNIiYygrEzlrBjvzqIikj9ogA4hbbN4pk9KYOikmOMm7GEvENHvS5JRKTGKACqkNayMbMmZLDn4FFumplFQZE6iIpI/aAACMCAdk14etwANuYe4uYXllJUcszrkkREqk0BEKALuibz2LX9yN5+gNtfXEbpMXUQFZHQpgA4Dd/o04oHr+zNgvV5TPn7CsrVQVREQpjOBD5N12e0Jb+wlN+/nUNiXDS/Gd1THURFJCQpAM7A94Z1Ir+whKc/2EJSfAz3XtTV65JERE6bAuAM3XdpGvmFpTwxbyNJcdFMPL+D1yWJiJwWBcAZMjMe/FZvDhaXcv8ba0mMi+aqAW28LktEJGCaBK6GyAjjsev6cn7n5vzoHyt5b+0er0sSEQmYAqCaGkRF8vS4AfRuncgdf/2UxZv3eV2SiEhAFAA1oGGDKGaNH0i7pvHcMjubVTsLvC5JRKRKCoAa0qRhDHMmDSIpPpqbZmWxKfew1yWJiJySAqAGtUz0dRCNMOPGGUvYlV/kdUkiIl9LAVDD2jdvyOyJGRw6Wsa4GUvYd1gdREUkOCkAakGPsxozc/xAducXcdOsLA4Vq4OoiAQfBUAtGdi+KU+OHUDOF4e4+YVsikvVQVREgosCoBYN75bCI9ecTda2/dz5188oUwdREQkiCoBaNqZva+4f04v31+3hR/9YqQ6iIhI01AqiDowb3I78IyU88t4GEuOi+eU3e6iDqIh4TgFQR+4c0Zn8olJmfLSVJvEx3DWyi9cliUiYC+gQkJmNMrP1ZrbJzO47xXZXm5kzs3T/coyZzTKzVWa2wsyG+dfHm9l/zCzHzNaY2UM1MpogZmb87LLuXNW/DY++t4HZi7d5XZKIhLkq3wGYWSQwHbgI2AksNbO5zrm1J22XANwFLKmw+hYA51xvM0sB3jKzgf77pjrnFphZDDDPzC51zr1V/SEFr4gI4/dX+TqI/vK1NSTGRTOmb2uvyxKRMBXIO4AMYJNzbotzrgR4GRhTyXYPAA8DxRXW9QDmATjncoF8IN05V+icW+BfXwJ8CoRFL+WoyAj+eH0/zunYjMmvrGB+jjqIiog3AgmA1sCOCss7/etOMLN+QKpz7o2T9l0BjDGzKDPrAAwAUk/aNwm4HH9QnMzMbjWzbDPLzsvLC6Dc4BcbHckzNw6ge6vGfO8vn5K1db/XJYlIGAokACr7uMqJzzKaWQQwDZhcyXYz8QVGNvAY8DFQVmHfKOAl4Ann3JbKHtw594xzLt05l56cnBxAuaEhITaa5ycMpE2TOCY9v5Q1u9VBVETqViABsJOv/tXeBthdYTkB6AVkmtk2YDAw18zSnXNlzrl7nHN9nXNjgCRgY4V9nwE2Ouceq84gQlWzRg2YM2kQCbFR3DQzi617j3hdkoiEkUACYCnQxcw6+CdsrwPmHr/TOVfgnGvunGvvnGsPfAKMds5l+z/t0xDAzC4Cyo5PHpvZb4FE4Ac1O6TQclZSHHNuHoRzMPa5JXxRoA6iIlI3qgwA51wZcCfwDrAOeMU5t8bM7jez0VXsngJ8ambrgB8D4wDMrA3wM3yTxJ+a2XIzu7ka4whpnZIb8cLEDAqKShk3I4v9R0q8LklEwoA5FzqtCdLT0112drbXZdSaT7bs46aZWaS1TODFWwbTqIHO0xOR6jOzZc659JPXqxdQEBncsRnTb+jP6t0H+e6cbI6WqYOoiNQeBUCQubBHC6Z+uw+LNu3j7peWq4OoiNQaBUAQurJfG351eQ/eXvMlP/3XKkLpMJ2IhA4dZA5SE87rQH5hKY/P20hiXDQ/vay7OoiKSI1SAASxH1zYhfzCEp79cCtJ8THcMbyz1yWJSD2iAAhiZsavLu9JQVEpf3hnPUnx0XxnUDuvyxKRekIBEOQiIow/fPtsDhWX8fN/r6ZxbDSXn32W12WJSD2gSeAQEB0ZwfTv9Gdgu6bc+8pyFm6oH03xRMRbCoAQERsdyXPj0+mSksBtc5axbLs6iIpI9SgAQkjj2GhmT8qgZWIsE2YtZd0XB70uSURCmAIgxDRv1IA5kzKIj4nixplZbN+nDqIicmYUACGoTZN4/nJzBmXHyhk7Ywl7DhZXvZOIyEkUACGqc0oCz0/IYP/hEm6ckUV+oTqIisjpUQCEsLNTk3j2xnS27j3CxOeXUlhSVvVOIiJ+CoAQd27n5vzxhn4s35HPd+csUwdREQmYAqAeuKRnSx66qg8fbtzLvX9bwbFyNY8TkarpTOB64pr0VA4WlfLb/6yjcVw0D17ZS83jROSUFAD1yM1DOnKgsITpCzaTFB/Nj0eleV2SiAQxBUA9M+XibuQXlvJk5maaxEdz6wWdvC5JRIKUAqCeMTPuH9OLgqJSHnwzh8S4aK4d2NbrskQkCCkA6qHICOPRa/pyqLiMn/xzFYlx0Yzq1crrskQkyOhTQPVUTFQET40dQL+2TbjrpeV8tHGv1yWJSJBRANRjcTGRzLxpIB2TG3LrnGw++/yA1yWJSBBRANRzifHRzJ6YQXJCAyY8v5QNew55XZKIBAkFQBhIaRzLXyYNIiYygnEzlrBjf6HXJYlIEFAAhInUpvHMmTSI4tJyxs1YQt6ho16XJCIeUwCEkW4tE5g1YSC5h45y48wsCopKvS5JRDykAAgz/ds24elxA9iUe4hJzy+lqETN40TClQIgDA3pkszj1/Xj088PcPuLyyg9Vu51SSLiAQVAmLqsdyt+d2VvFqzPY/IrKyhXB1GRsKMzgcPY9RltKSgq5aG3fC0j7h/TUx1ERcKIAiDM3Ta0EwcKS3h64RaaxEdz78XdvC5JROqIAkC4b1QaBYWlPDF/E4nxMUw6v4PXJYlIHVAACGbG767szcHiUh54Yy1JcdFcNaCN12WJSC1TAAjg6yA67dq+HCrO5kf/WElCbBQX92zpdVkiUov0KSA5oUFUJE+NHUDv1onc+dJnLN68z+uSRKQWKQDkKxo2iOL5CQNp3yyeW2Zns3JnvtcliUgtUQDI/0iKj2HOpEEkxUczftZSNuUe9rokEakFCgCpVAt/B9EIM8bNWMKu/CKvSxKRGhZQAJjZKDNbb2abzOy+U2x3tZk5M0v3L8eY2SwzW2VmK8xsWIVtB/jXbzKzJ0xnIAWd9s0bMmdSBoePljHuuSXsPawOoiL1SZUBYGaRwHTgUqAHcL2Z9ahkuwTgLmBJhdW3ADjnegMXAY+Y2fHHfBK4Feji/xp15sOQ2tK9VWNmjR/I7oIixs/K4lCxOoiK1BeBvAPIADY557Y450qAl4ExlWz3APAwUFxhXQ9gHoBzLhfIB9LNrBXQ2Dm32DnngNnAFWc+DKlN6e2b8tTYAeR8cYibX8imuFQdREXqg0ACoDWwo8LyTv+6E8ysH5DqnHvjpH1XAGPMLMrMOgADgFT//jtP9T0rfO9bzSzbzLLz8vICKFdqw7BuKTx6bV+ytu3nzr9+qg6iIvVAIAFQ2bH5E60j/Yd0pgGTK9luJr4X92zgMeBjoKyq7/mVlc4945xLd86lJycnB1Cu1JbRZ5/F/WN68f66XH786kp1EBUJcYGcCbwT31/tx7UBdldYTgB6AZn+edyWwFwzG+2cywbuOb6hmX0MbAQO+L/P131PCVLjBrejoLCEqe9uoHFcNL+6vIc6iIqEqEACYCnQxX8IZxdwHXDD8TudcwVA8+PLZpYJTHHOZZtZPGDOuSNmdhFQ5pxb69/ukJkNxjdpfCPwxxoak9SyO4Z3Jr+wlOc+2kqT+BjuvrCL1yWJyBmoMgCcc2VmdifwDhAJzHTOrTGz+4Fs59zcU+yeArxjZuX4wmNchfu+BzwPxAFv+b8kBJgZP/tGd/KLSpn2/gaS4qO56dz2XpclIqfJfB/CCQ3p6ekuOzvb6zLEr+xYObe/+Cnvrt3DY9f25Yp+lc7ji4jHzGyZcy795PU6E1jOWFRkBE9c349zOjZj8t9XMD9nj9clichpUABItcRGR/LsTen0PKsx3/vLp2Rt3e91SSISIAWAVFujBlE8PyGDNk3imPT8UlbvKvC6JBEJgAJAakTThjH85eZBNI6L5qaZWWzJUwdRkWCnAJAa0yoxjjmTMgAYNyOLLwrUQVQkmCkApEZ1TG7ECxMzOFhUyrgZWew/UuJ1SSLyNRQAUuN6tU7kuZvS2bG/kAmzsjh8tMzrkkSkEgoAqRWDOjbjz9/pz+rdB7l1tjqIigQjBYDUmpHdW/DIt8/m4837uOulzyhTB1GRoKIAkFp1Rb/W/PryHry7dg8/+ecqQunMc5H6LpBmcCLVMv68DuQXlfLY+xtJio/mp5d1VwdRkSCgAJA6cffILuQXlvLsh1tJio/hjuGdvS5JJOwpAKROmBm//GYPCopK+cM760mMi2bs4HZelyUS1hQAUmciIoyHr+7DoeJSfvHaahLjorn87LO8LkskbGkSWOpUdGQEf7qhPwPbN+UHf1vOr+euIb9QJ4uJeEEBIHUuNjqSGTelc31GKrMXb2PY1ExmL96mj4mK1DEFgHgiITaa317RmzfvHkKPVo355WtruOyJD/lwY57XpYmEDQWAeCqtZWNevHkQT48bQHFpOeNmZHHzC9ls3XvE69JE6j0FgHjOzLikZ0veu/cC7rs0jU+27OPiaQt58M11HCwu9bo8kXpLASBBo0FUJLcN7cT8KUP5Vr82PPvhFkZMzeTlrM85Vq4ziEVqmgJAgk5KQiy/v7oPr995Ph2aN+S+f67i8j9+xJIt+7wuTaReUQBI0OrVOpFXvnsOf7qhHwVFpVz7zCfc/uIyduwv9Lo0kXpBASBBzcz4Zp+zmDd5KPde1JUFOXmMfHQhf3gnhyO6zoBItSgAJCTERkdy18guzJ8ylG/0bsX0BZsZPjWTfyzbSbnmB0TOiAJAQkqrxDimXduXf95+Lq2S4pj89xVc+eTHLNt+wOvSREKOAkBCUv+2TfjX987l0WvO5suCIq568mN+8PJnuhC9yGlQAEjIiogwvtW/DfMnD+PO4Z15c/WXjJi6kMff30hRiS5BKVIVBYCEvIYNophySTfm3TuUEWkpTHt/Axc+upDXV+zWFchETkEBIPVGatN4pn+nP3+7dTCJcdF8/6XPuObpxazaWeB1aSJBSQEg9c6gjs14/fvn89C3erN17xFGT/+IH726gtxDxV6XJhJUFABSL0VGGNdltGX+lGHcMqQj//psFyOmLuTJzM0cLdP8gAgoAKSeaxzruwj9u/cMZXDHZvz+7RwuevQD3lnzpeYHJOwpACQsdGjekOduSmfOpAxioyP47pxlfOe5JeR8edDr0kQ8owCQsDKkSzJv3jWE+8f0ZO0XB7ns8Q/5+b9Xsf+ILksp4UcBIGEnKjKCG89pT+aUYdx4TnteytrBsD8sYMZHWynVZSkljCgAJGwlxcfw69E9efvuIfRt24QH3ljLJY99wIL1uV6XJlInFAAS9rq0SOCFCQOZOT4dHEyYtZTxs7LYlHvY69JEapUCQARf2+kRaS14+wcX8PNvdGfZ9gOMeuwDfvP6GgoKdVlKqZ8UACIVxERFcPOQjmROGcY1A1N54eNtDJu6gDmLt1Gm+QGpZwIKADMbZWbrzWyTmd13iu2uNjNnZun+5Wgze8HMVpnZOjP7SYVt7zGzNWa22sxeMrPY6g9HpGY0a9SAB6/szRvfH0K3lgn84rU1fOOJj1i0aa/XpYnUmCoDwMwigenApUAP4Hoz61HJdgnAXcCSCqu/DTRwzvUGBgDfNbP2Ztbav226c64XEAlcV93BiNS0Hmc15qVbBvPU2P4UlpbxneeWcMvsbLbtPeJ1aSLVFsg7gAxgk3Nui3OuBHgZGFPJdg8ADwMVG644oKGZRQFxQAlw/MybKCDOf188sPvMhiBSu8yMUb1a8d49Q/nhJd1YtGkvF0/7gP97ax2HijU/IKErkABoDeyosLzTv+4EM+sHpDrn3jhp31eBI8AXwOfAVOfcfufcLmCqf90XQIFz7t3KHtzMbjWzbDPLzsvLC2RMIrUiNjqSO4Z3JnPKMEb3PYunF25h+NSF/G3p5xzTZSklBAUSAFbJuhO/7WYWAUwDJleyXQZwDDgL6ABMNrOOZtYE37uIDv77GprZ2Moe3Dn3jHMu3TmXnpycHEC5IrUrpXEsU799Nq/dcR7tmsXz43+sYsz0j8jaut/r0kROSyABsBNIrbDchq8erkkAegGZZrYNGAzM9U8E3wC87Zwrdc7lAouAdOBCYKtzLs85Vwr8Ezi3uoMRqUtnpybx6m3n8Ph1fdl3uIRrnl7MHX/9lJ0HCr0uTSQggQTAUqCLmXUwsxh8k7Vzj9/pnCtwzjV3zrV3zrUHPgFGO+ey8R3iGWE+DfGFQ45//WAzizczA0YC62p0ZCJ1wMwY07c18ycP4+6RXZi3bg8jH1nIo++up7CkzOvyRE6pygBwzpUBdwLv4HuRfsU5t8bM7jez0VXsPh1oBKzGFySznHMrnXNL8M0PfAqs8tfxzJkPQ8RbcTGR3HNRV+ZPHsYlPVvyxPxNjJi6kH99tpNyzQ9IkLJQ6omenp7usrOzvS5DpErLtu/nN6+vZeXOAvq1TeJXl/ekb2qS12VJmDKzZc659JPX60xgkVowoF1T/n37eUz99tnsPFDEFdMXce/flvNlgS5LKcFDASBSSyIijKsHtGHBlGHcPqwTb6z6guFTM/nT/I0Ul+qylOI9BYBILWvUIIofjUpj3r1DGdYtmanvbmDkIwv5z8ovdFlK8ZQCQKSOpDaN58mxA3jplsEkxEZxx18/5dpnPmH1rgKvS5MwpQAQqWPndGrGf+4awoNX9mZT7mEu/9NH3PePleQdOup1aRJmFAAiHoiMMG4Y1JYFU4Yx6bwOvLpsJ8OnZvLMB5spKVPbaakbCgARDyXGRfPzb/bg3XsuYFCHpjz4Zg4XT1vIe2v3aH5Aap0CQCQIdExuxIzxA3lhYgZRkRHcMjubcTOyWP/lIa9Lk3pMASASRIZ2Teatu4fw68t7sGpXAZc+/gG/+PdqDhwp8bo0qYcUACJBJjoygvHndSBzyjDGDm7HX7M+Z9jUTGYt2kqpLkspNUgBIBKkmjSM4f4xvXjr7iH0aZPIb15fy6WPf0jm+lyvS5N6QgEgEuS6tkhg9sQMnr0xnbJj5YyftZSJzy9lc95hr0uTEKcAEAkBZsZFPVrwzj0X8NPL0li6dT+XTPuAB95YS0GRLkspZ0YBIBJCGkRFcusFnZg/ZRhXD2jDzEVbGT41kxeXbNdlKeW0KQBEQlByQgMeuqoPr995Pp1TGvGzf63mG098yMeb93pdmoQQBYBICOvVOpG/3TqYP3+nP4eKy7jh2SXcNmcZn+/TZSmlagoAkRBnZlzWuxXzJg9lysVd+WBjHhc+upDfv53D4aO6LKV8PQWASD0RGx3JnSO6MH/yML7ZpxVPZm5m+NRM/p69Q5ellEopAETqmZaJsTx6bV/+dfu5tGkSxw9fXckVf15E9rb9XpcmQUYBIFJP9WvbhH/cdi6PXduX3INHufqpxXz/pc/YlV/kdWkSJBQAIvVYRIRxRb/WzJ8ylLtGdObdNV8y8pFMpr23gaISXZYy3CkARMJAfEwU917cjXmTh3Jh9xY8Pm8jIx7J5LXlu9R2OowpAETCSJsm8fzphv688t1zaNYohrtfXs7VTy1mxY58r0sTDygARMJQRoemvHbH+Tx8VR+27zvCmOmLmPzKCnIPFntdmtShKK8LEBFvREYY1wxM5dLeLfnTgk3M+mgbb63+gqv6t2FE9xTO6diM2OhIr8uUWmShdPwvPT3dZWdne12GSL20fd8Rpr67gffX7qGo9Bhx0ZGc17kZI9JaMDwtmVaJcV6XKGfIzJY559JPXq93ACICQLtmDfnj9f0oLj3GJ1v2MT8nl/k5uby/znf9gR6tGjMiLYUR3VM4u00SkRHmccVSXXoHICJfyznHxtzDvjBYl8uyzw9wrNzRtGEMw7omM6J7CkO6JJMYF+11qXIKX/cOQAEgIgHLLyxh4YY8FuTkkrkhj/zCUiIjjIHtm/jeHaS1oFNyQ8z07iCYKABEpEYdK3d89vmBE4eKcr48BEDbpvH+MEhhUMemNIjSRLLXFAAiUqt25RcxPyeXBTm5LNq0l6Nl5cTHRHJ+5+aM7J7C8G4ppDSO9brMsKQAEJE6U1RyjMVb9jJvnS8Qdhf4zi/o1boxI9JaMCIthT6tE4nQRHKdUACIiCecc+R8eejEoaLPPj9AuYPmjRowrFsyI9NSOL9LcxJiNZFcWxQAIhIUDhzxTSTPy8ll4fpcDhaXER1pZHRoyvBuKYzs3oIOzRt6XWa9ogAQkaBTdqycZdsPMH+972OmG3MPA9ChecMTE8kD2zclJkpda6pDASAiQW/H/sITh4oWb9lHSVk5jRpEMaRLc0akpTCsWwrJCQ28LjPkKABEJKQUlpSxaNM+5ufsYX5OLnsOHgXg7NQkRnTzvTvoeVZjTSQHQAEgIiHLOcea3QdZkJPLvJxcVuzMxzlISWjA8G6+9hTnd25OwwbqblMZBYCI1Bt7Dx9l4fo85ufk8sGGPA4dLSMmMoJBHZsyIi2FkWktaNss3usyg0a1AsDMRgGPA5HAc865h75mu6uBvwMDnXPZZhYNPAf0x9d4brZz7v/82yb57+sFOGCic27xqepQAIjIyUqPlbN02/4T7w625B0BoFNyQ0Z2b8Hwbimkt29CdGT4TiSfcQCYWSSwAbgI2AksBa53zq09absE4D9ADHCnPwBuAEY7564zs3hgLTDMObfNzF4APnTOPWdmMUC8c+6UlyVSAIhIVbbtPeI7I3l9Lp9s2UfpMUdCbBQXdPWdczCsWwpNG8Z4XWadqk476Axgk3Nui/8bvQyMwfdiXtEDwMPAlArrHNDQzKKAOKAEOGhmjYELgPEAzrkS/30iItXSvnlDJp7fgYnnd+Dw0TI+2riX+Tl7WLA+j/+s/AIz6JeadKJ5XfdWCWHbvC6QAGgN7KiwvBMYVHEDM+sHpDrn3jCzigHwKr6w+AKIB+5xzu03s75AHjDLzM4GlgF3O+eOnPzgZnYrcCtA27ZtAx6YiEijBlGM6tWSUb1aUl7uWL274ES/oqnvbmDquxtolRjLsG4pjExL4bzOzYmLCZ/mdYEEQGXReOK4kZlFANPw/zV/kgzgGHAW0AT40Mze9z9uf+D7zrklZvY4cB/wi/95IOeeAZ4B3yGgAOoVEfkfERFGnzZJ9GmTxA8u7EruoWIy1+cxf10uc5fv4qWsz4mJiuDcTs0YkeZrXpfatH5PJAcSADuB1ArLbYDdFZYT8E3kZvrfRrUE5prZaOAG4G3nXCmQa2aLgHTgA2Cnc26J/3u8ii8ARETqREpCLNekp3JNeipHy46xdOvx1tZ7+OVra4A1dG3R6ETzuv5tk4iqZxPJgUwCR+GbBB4J7MI3CXyDc27N18FYvrAAAAadSURBVGyfCUzxTwL/GEgDJuI7BLQUuM45t9LMPgRuds6tN7NfAw2dcz88VS2aBBaRurAl7/CJM5Kztu6nrNyRGBfN0K7JjOyewtCuySTFh85E8hlPAjvnyszsTuAdfB8DnemcW2Nm9wPZzrm5p9h9OjALWI3vUNIs59xK/33fB170fwJoCzDhtEYkIlJLOiY3omNyI24e0pGDxaV8tNHX2jpzfS5zV+wmwmBAuyYM959z0LVFo5CcSNaJYCIiASovd6zYmc+CnFzmr89l9a6DALROijvRvO6cTs2IjQ6uiWSdCSwiUsP2HCw+cQLaok17KSw5Rmx0BOd1as5wfyCclRTndZkKABGR2lRceowlW4+fkbyHHfuLAEhrmcDI7r4w6JvahEgPmtcpAERE6ohzjs15h5m3zjeRnL39AMfKHU3ioxnm72R6QddkEuPq5ipoCgAREY8UFJbywUZf87rM9bkcKCwlMsJIb9fE17yuewqdkmtvIlkBICISBI6VO5bvOH7OQR7rvvBNJKc2jWNkWguGp6UwqEPTGp1IVgCIiASh3flFLPBfEnPR5r0Ul5YTHxPJeZ2bMzItheFpKbRoHFutx1AAiIgEueLSYyzevO/ESWi78n0TyT3PaszsiRk0a3Rml8OsTjdQERGpA7HRkQz3/9V/v3Ns2HOYeTl7WLEjv1ZaWCsARESCkJnRrWUC3Vom1Npj1K/ORiIiEjAFgIhImFIAiIiEKQWAiEiYUgCIiIQpBYCISJhSAIiIhCkFgIhImAqpVhBmlgdsP8PdmwN7a7AcL9WXsdSXcYDGEqzqy1iqO452zrnkk1eGVABUh5llV9YLIxTVl7HUl3GAxhKs6stYamscOgQkIhKmFAAiImEqnALgGa8LqEH1ZSz1ZRygsQSr+jKWWhlH2MwBiIjIV4XTOwAREalAASAiEqZCPgDMbJSZrTezTWZ2XyX332tma81spZnNM7N2Fe67ycw2+r9uqtvK/1c1x3LMzJb7v+bWbeX/K4Cx3GZmq/z1fmRmPSrc9xP/fuvN7JK6rfx/nelYzKy9mRVVeF6eqvvqv1LnKcdRYburzcyZWXqFdSH1nFTY7itjCbbnxF9TVb9f480sr0LNN1e4r3qvYc65kP0CIoHNQEcgBlgB9Dhpm+FAvP/294C/+W83Bbb4/23iv90kFMfiXz7s9fNxmmNpXOH2aOBt/+0e/u0bAB383ycyRMfSHljt9fMR6Dj82yUAHwCfAOmh+pycYixB85ycxu/XeOBPlexb7dewUH8HkAFscs5tcc6VAC8DYypu4Jxb4Jwr9C9+ArTx374EeM85t985dwB4DxhVR3VXpjpjCTaBjOVghcWGwPFPI4wBXnbOHXXObQU2+b+fV6ozlmBS5Tj8HgAeBoorrAu558SvsrEEm0DHUplqv4aFegC0BnZUWN7pX/d1JgFvneG+ta06YwGINbNsM/vEzK6ojQJPQ0BjMbM7zGwzvv+kd53OvnWoOmMB6GBmn5nZQjMbUrulnlKV4zCzfkCqc+6N0923jlVnLBA8zwkE/rO9yn/o91UzSz3Nfb9WqAeAVbKu0r++zGwskA784XT3rSPVGQtAW+c7VfwG4DEz61TzJQYsoLE456Y75zoBPwZ+fjr71qHqjOULfM9LP+Be4K9m1rjWKj21U47DzCKAacDk093XA9UZSzA9JxDYz/Z1oL1zrg/wPvDCaex7SqEeADuB1ArLbYDdJ29kZhcCPwNGO+eOns6+dag6Y8E5t9v/7xYgE+hXm8VW4XR/ti8Dx9+1hOTzUsGJsfgPmezz316G71hv11qqsypVjSMB6AVkmtk2YDAw1z95GmrPydeOJcieEwjgZ+uc21fh//qzwIBA962S15Mg1ZxAicI38dGB/06g9Dxpm374nuQulUygbMU3edLEf7tpiI6lCdDAf7s5sJFKJsWCbCxdKty+HMj23+7JVycct+DthGN1xpJ8vHZ8k3y7vPodC2QcJ22fyX8nTkPuOTnFWILmOTmN369WFW5fCXziv13t1zBPBl3DP8DLgA3+F8af+dfdj+8vZPC9ZdoDLPd/za2w70R8E1qbgAmhOhbgXGCV/5dnFTApBMbyOLDGP44FFX/p8b3D2QysBy4N1bEAV/nXrwA+BS4P5nGctO2JF81QfE6+bizB9pwE+Pv1fxVqXgCkVdi3Wq9hagUhIhKmQn0OQEREzpACQEQkTCkARETClAJARCRMKQBERMKUAkBEJEwpAEREwtT/AxplEiHHTom7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "test_means = grid_search.cv_results_['mean_test_score']\n",
    "test_stds = grid_search.cv_results_['std_test_score']\n",
    "x_axis = colsample_bytree_s\n",
    "plt.plot(x_axis,-test_means)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 3 candidates, totalling 9 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=4)]: Done   4 out of   9 | elapsed:  7.8min remaining:  9.8min\n",
      "[Parallel(n_jobs=4)]: Done   6 out of   9 | elapsed: 13.6min remaining:  6.8min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4854507775458169\n",
      "{'learning_rate': 0.1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done   9 out of   9 | elapsed: 20.5min finished\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "#          'learning_rate':0.1,\n",
    "         'max_bin':127,\n",
    "         'bagging_freq':1,\n",
    "         'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75,\n",
    "         'min_child_samples':40,\n",
    "         'subsample':0.7,\n",
    "          'colsample_bytree':0.5\n",
    "         }\n",
    "learning_rate_s = np.logspace(-3,-1,3)\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "tuned_parameters = dict(learning_rate=learning_rate_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='neg_log_loss',verbose=5,refit=False)\n",
    "grid_search.fit(x_train,y_train)\n",
    "\n",
    "print(-grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.001 0.01  0.1  ]\n"
     ]
    }
   ],
   "source": [
    "print(np.logspace(-3,-1,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators:  3808\n",
      "best cv score:  0.47616300022169217\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.01,\n",
    "         'max_bin':127,\n",
    "         'bagging_freq':1,\n",
    "#          'n_estimators':n_estimators,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75,\n",
    "         'min_child_samples':40,\n",
    "         'subsample':0.7,\n",
    "          'colsample_bytree':0.5\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params,x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello\n"
     ]
    }
   ],
   "source": [
    "print('hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(bagging_freq=1, boosting_type='gbdt', class_weight=None,\n",
       "               colsample_bytree=0.5, importance_type='split',\n",
       "               learning_rate=0.01, max_bin=127, max_depth=7,\n",
       "               min_child_samples=40, min_child_weight=0.001, min_split_gain=0.0,\n",
       "               n_estimators=3808, n_jobs=4, num_class=9, num_leaves=75,\n",
       "               objective='multiclass', random_state=None, reg_alpha=0.0,\n",
       "               reg_lambda=0.0, silent=False, subsample=0.7,\n",
       "               subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type':'gbdt',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.01,\n",
    "         'max_bin':127,\n",
    "         'bagging_freq':1,\n",
    "         'n_estimators':3808,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':75,\n",
    "         'min_child_samples':40,\n",
    "         'subsample':0.7,\n",
    "          'colsample_bytree':0.5\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,**params)\n",
    "lg.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello\n"
     ]
    }
   ],
   "source": [
    "print('hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 保存模型\n",
    "cPickle.dump(lg,open(\"Otto/lg_model.pkl\",'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 测试特征数据转化\n",
    "test = pd.read_csv('Otto/test.csv')\n",
    "test.head()\n",
    "test_id = test['id']\n",
    "x_test = test.drop(['id',],axis = 1)\n",
    "columns_org = x_test.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>feat_10_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.183692</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.412891</td>\n",
       "      <td>0.052117</td>\n",
       "      <td>0.841374</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.067827</td>\n",
       "      <td>0.078237</td>\n",
       "      <td>0.442733</td>\n",
       "      <td>0.494155</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.122685</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.06132</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.058936</td>\n",
       "      <td>0.571732</td>\n",
       "      <td>0.046531</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078015</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.069945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.044685</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.135478</td>\n",
       "      <td>0.033424</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.047748</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.043484</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.058626</td>\n",
       "      <td>0.079708</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.159172</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.547803</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 93 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0      0.000000      0.000000      0.000000      0.000000           0.0   \n",
       "1      0.067827      0.078237      0.442733      0.494155           0.0   \n",
       "2      0.000000      0.058936      0.571732      0.046531           0.0   \n",
       "3      0.000000      0.000000      0.000000      0.044685           0.0   \n",
       "4      0.047748      0.000000      0.000000      0.043484           0.0   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  feat_10_tfidf  ...  \\\n",
       "0           0.0      0.000000      0.000000           0.0       0.183692  ...   \n",
       "1           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "2           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "3           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "4           0.0      0.058626      0.079708           0.0       0.159172  ...   \n",
       "\n",
       "   feat_84_tfidf  feat_85_tfidf  feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  \\\n",
       "0            0.0       0.000000       0.412891       0.052117       0.841374   \n",
       "1            0.0       0.000000       0.000000       0.000000       0.000000   \n",
       "2            0.0       0.000000       0.000000       0.000000       0.078015   \n",
       "3            0.0       0.135478       0.033424       0.000000       0.000000   \n",
       "4            0.0       0.000000       0.000000       0.000000       0.000000   \n",
       "\n",
       "   feat_89_tfidf  feat_90_tfidf  feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0       0.000000            0.0       0.000000        0.00000       0.000000  \n",
       "1       0.122685            0.0       0.000000        0.06132       0.000000  \n",
       "2       0.000000            0.0       0.000000        0.00000       0.069945  \n",
       "3       0.000000            0.0       0.000000        0.00000       0.000000  \n",
       "4       0.000000            0.0       0.547803        0.00000       0.000000  \n",
       "\n",
       "[5 rows x 93 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf = cPickle.load(open('Otto/tfidf.pkl','rb'))\n",
    "x_test_tfidf = tfidf.fit_transform(x_test).toarray()\n",
    "feat_names = columns_org + \"_tfidf\"\n",
    "x_test_tfidf = pd.DataFrame(columns = feat_names, data = x_test_tfidf)\n",
    "x_test_tfidf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "ms_tfidf = cPickle.load(open('Otto/MinMaxScaler_tfidf.pkl','rb'))\n",
    "feat_name_tfidf = x_test_tfidf.columns\n",
    "x_test_tfidf = ms_tfidf.fit_transform(x_test_tfidf)\n",
    "test_tfidf = pd.concat([test_id,pd.DataFrame(columns = feat_name_tfidf,data = x_test_tfidf)],axis = 1)\n",
    "test_tfidf.head()\n",
    "test_tfidf.to_csv('Otto/Otto_FE_test_tfidf.csv',index=False,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.412891</td>\n",
       "      <td>0.058633</td>\n",
       "      <td>0.841374</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.068521</td>\n",
       "      <td>0.085005</td>\n",
       "      <td>0.485557</td>\n",
       "      <td>0.516251</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.144047</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.06132</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.064034</td>\n",
       "      <td>0.627034</td>\n",
       "      <td>0.048611</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078015</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.070429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.046683</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.135478</td>\n",
       "      <td>0.033424</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.048236</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.045428</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.063757</td>\n",
       "      <td>0.079708</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.547803</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 94 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0   1      0.000000      0.000000      0.000000      0.000000           0.0   \n",
       "1   2      0.068521      0.085005      0.485557      0.516251           0.0   \n",
       "2   3      0.000000      0.064034      0.627034      0.048611           0.0   \n",
       "3   4      0.000000      0.000000      0.000000      0.046683           0.0   \n",
       "4   5      0.048236      0.000000      0.000000      0.045428           0.0   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  ...  feat_84_tfidf  \\\n",
       "0           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "1           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "2           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "3           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "4           0.0      0.063757      0.079708           0.0  ...            0.0   \n",
       "\n",
       "   feat_85_tfidf  feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  \\\n",
       "0       0.000000       0.412891       0.058633       0.841374       0.000000   \n",
       "1       0.000000       0.000000       0.000000       0.000000       0.144047   \n",
       "2       0.000000       0.000000       0.000000       0.078015       0.000000   \n",
       "3       0.135478       0.033424       0.000000       0.000000       0.000000   \n",
       "4       0.000000       0.000000       0.000000       0.000000       0.000000   \n",
       "\n",
       "   feat_90_tfidf  feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0            0.0       0.000000        0.00000       0.000000  \n",
       "1            0.0       0.000000        0.06132       0.000000  \n",
       "2            0.0       0.000000        0.00000       0.070429  \n",
       "3            0.0       0.000000        0.00000       0.000000  \n",
       "4            0.0       0.547803        0.00000       0.000000  \n",
       "\n",
       "[5 rows x 94 columns]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_tfidf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.412891</td>\n",
       "      <td>0.058633</td>\n",
       "      <td>0.841374</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.068521</td>\n",
       "      <td>0.085005</td>\n",
       "      <td>0.485557</td>\n",
       "      <td>0.516251</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.144047</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.06132</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.064034</td>\n",
       "      <td>0.627034</td>\n",
       "      <td>0.048611</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078015</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.070429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.046683</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.135478</td>\n",
       "      <td>0.033424</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.048236</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.045428</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.063757</td>\n",
       "      <td>0.079708</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.547803</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 94 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0   1      0.000000      0.000000      0.000000      0.000000           0.0   \n",
       "1   2      0.068521      0.085005      0.485557      0.516251           0.0   \n",
       "2   3      0.000000      0.064034      0.627034      0.048611           0.0   \n",
       "3   4      0.000000      0.000000      0.000000      0.046683           0.0   \n",
       "4   5      0.048236      0.000000      0.000000      0.045428           0.0   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  ...  feat_84_tfidf  \\\n",
       "0           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "1           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "2           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "3           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "4           0.0      0.063757      0.079708           0.0  ...            0.0   \n",
       "\n",
       "   feat_85_tfidf  feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  \\\n",
       "0       0.000000       0.412891       0.058633       0.841374       0.000000   \n",
       "1       0.000000       0.000000       0.000000       0.000000       0.144047   \n",
       "2       0.000000       0.000000       0.000000       0.078015       0.000000   \n",
       "3       0.135478       0.033424       0.000000       0.000000       0.000000   \n",
       "4       0.000000       0.000000       0.000000       0.000000       0.000000   \n",
       "\n",
       "   feat_90_tfidf  feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0            0.0       0.000000        0.00000       0.000000  \n",
       "1            0.0       0.000000        0.06132       0.000000  \n",
       "2            0.0       0.000000        0.00000       0.070429  \n",
       "3            0.0       0.000000        0.00000       0.000000  \n",
       "4            0.0       0.547803        0.00000       0.000000  \n",
       "\n",
       "[5 rows x 94 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 进行测试\n",
    "test = pd.read_csv('Otto/Otto_FE_test_tfidf.csv')\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>feat_10_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.188847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.412891</td>\n",
       "      <td>0.058633</td>\n",
       "      <td>0.841374</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.068521</td>\n",
       "      <td>0.085005</td>\n",
       "      <td>0.485557</td>\n",
       "      <td>0.516251</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.144047</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.06132</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.064034</td>\n",
       "      <td>0.627034</td>\n",
       "      <td>0.048611</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078015</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.070429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.046683</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.135478</td>\n",
       "      <td>0.033424</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.048236</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.045428</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.063757</td>\n",
       "      <td>0.079708</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.163639</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.547803</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 93 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0      0.000000      0.000000      0.000000      0.000000           0.0   \n",
       "1      0.068521      0.085005      0.485557      0.516251           0.0   \n",
       "2      0.000000      0.064034      0.627034      0.048611           0.0   \n",
       "3      0.000000      0.000000      0.000000      0.046683           0.0   \n",
       "4      0.048236      0.000000      0.000000      0.045428           0.0   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  feat_10_tfidf  ...  \\\n",
       "0           0.0      0.000000      0.000000           0.0       0.188847  ...   \n",
       "1           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "2           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "3           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "4           0.0      0.063757      0.079708           0.0       0.163639  ...   \n",
       "\n",
       "   feat_84_tfidf  feat_85_tfidf  feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  \\\n",
       "0            0.0       0.000000       0.412891       0.058633       0.841374   \n",
       "1            0.0       0.000000       0.000000       0.000000       0.000000   \n",
       "2            0.0       0.000000       0.000000       0.000000       0.078015   \n",
       "3            0.0       0.135478       0.033424       0.000000       0.000000   \n",
       "4            0.0       0.000000       0.000000       0.000000       0.000000   \n",
       "\n",
       "   feat_89_tfidf  feat_90_tfidf  feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0       0.000000            0.0       0.000000        0.00000       0.000000  \n",
       "1       0.144047            0.0       0.000000        0.06132       0.000000  \n",
       "2       0.000000            0.0       0.000000        0.00000       0.070429  \n",
       "3       0.000000            0.0       0.000000        0.00000       0.000000  \n",
       "4       0.000000            0.0       0.547803        0.00000       0.000000  \n",
       "\n",
       "[5 rows x 93 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_id = test['id']\n",
    "x_test = test.drop(['id'],axis =1)\n",
    "x_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#保存特征名字后边备用\n",
    "feat_names = x_test.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 稀疏存储，模型训练更快\n",
    "from scipy.sparse import csr_matrix\n",
    "x_test = csr_matrix(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(144368, 9)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = cPickle.load(open('Otto/lg_model.pkl','rb'))\n",
    "y_test_pred = model.predict_proba(x_test)\n",
    "y_test_pred.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Class_1</th>\n",
       "      <th>Class_2</th>\n",
       "      <th>Class_3</th>\n",
       "      <th>Class_4</th>\n",
       "      <th>Class_5</th>\n",
       "      <th>Class_6</th>\n",
       "      <th>Class_7</th>\n",
       "      <th>Class_8</th>\n",
       "      <th>Class_9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000636</td>\n",
       "      <td>0.081427</td>\n",
       "      <td>0.136715</td>\n",
       "      <td>0.771764</td>\n",
       "      <td>4.569049e-06</td>\n",
       "      <td>0.000841</td>\n",
       "      <td>0.008170</td>\n",
       "      <td>0.000309</td>\n",
       "      <td>0.000134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.003305</td>\n",
       "      <td>0.010247</td>\n",
       "      <td>0.002022</td>\n",
       "      <td>0.000652</td>\n",
       "      <td>1.077896e-04</td>\n",
       "      <td>0.821478</td>\n",
       "      <td>0.001653</td>\n",
       "      <td>0.157738</td>\n",
       "      <td>0.002797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000030</td>\n",
       "      <td>0.000057</td>\n",
       "      <td>0.000089</td>\n",
       "      <td>0.000018</td>\n",
       "      <td>4.631967e-07</td>\n",
       "      <td>0.999293</td>\n",
       "      <td>0.000054</td>\n",
       "      <td>0.000420</td>\n",
       "      <td>0.000038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000771</td>\n",
       "      <td>0.556247</td>\n",
       "      <td>0.424164</td>\n",
       "      <td>0.017179</td>\n",
       "      <td>3.485355e-06</td>\n",
       "      <td>0.000236</td>\n",
       "      <td>0.000239</td>\n",
       "      <td>0.000427</td>\n",
       "      <td>0.000733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.221002</td>\n",
       "      <td>0.001138</td>\n",
       "      <td>0.001390</td>\n",
       "      <td>0.000130</td>\n",
       "      <td>6.138591e-06</td>\n",
       "      <td>0.009678</td>\n",
       "      <td>0.004431</td>\n",
       "      <td>0.074508</td>\n",
       "      <td>0.687717</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id   Class_1   Class_2   Class_3   Class_4       Class_5   Class_6  \\\n",
       "0   1  0.000636  0.081427  0.136715  0.771764  4.569049e-06  0.000841   \n",
       "1   2  0.003305  0.010247  0.002022  0.000652  1.077896e-04  0.821478   \n",
       "2   3  0.000030  0.000057  0.000089  0.000018  4.631967e-07  0.999293   \n",
       "3   4  0.000771  0.556247  0.424164  0.017179  3.485355e-06  0.000236   \n",
       "4   5  0.221002  0.001138  0.001390  0.000130  6.138591e-06  0.009678   \n",
       "\n",
       "    Class_7   Class_8   Class_9  \n",
       "0  0.008170  0.000309  0.000134  \n",
       "1  0.001653  0.157738  0.002797  \n",
       "2  0.000054  0.000420  0.000038  \n",
       "3  0.000239  0.000427  0.000733  \n",
       "4  0.004431  0.074508  0.687717  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out_df = pd.DataFrame(y_test_pred)\n",
    "columns = np.empty(9,dtype=object)\n",
    "for i in range(9):\n",
    "    columns[i] = 'Class_'+str(i+1)\n",
    "out_df.columns = columns\n",
    "out_df = pd.concat([test_id,out_df],axis = 1)\n",
    "out_df.to_csv('Otto/LightGBM_tfidf.csv',index=False)\n",
    "out_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "提交kaggle获取的分数是0.45641"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 6. 采用tfidf特征，采用LightGBM（goss）完成Otto商品分类，尽可能将超参数调到最优，并用该模型对测试数据进行测试，提交Kaggle网站，提交排名。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='goss', class_weight=None, colsample_bytree=0.4,\n",
       "               importance_type='split', learning_rate=0.01, max_bin=127,\n",
       "               max_depth=7, min_child_samples=10, min_child_weight=0.001,\n",
       "               min_split_gain=0.0, n_estimators=2674, n_jobs=4, num_class=9,\n",
       "               num_leaves=70, objective='multiclass', random_state=None,\n",
       "               reg_alpha=0.0, reg_lambda=0.0, silent=False, subsample=1.0,\n",
       "               subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 进行GOSS的训练，过程和gbdt一样直接使用参数获取模型\n",
    "params = {'boosting_type':'goss',\n",
    "         'objective':'multiclass',\n",
    "         'n_jobs':4,\n",
    "         'learning_rate':0.01,\n",
    "         'max_bin':127,\n",
    "         'n_estimators':2674,\n",
    "         'max_depth':7,\n",
    "         'num_class':9,\n",
    "         'num_leaves':70,\n",
    "         'min_child_samples':10,\n",
    "         'colsample_bytree':0.4\n",
    "         }\n",
    "lg_goss = LGBMClassifier(silent=False,**params)\n",
    "lg_goss.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello\n"
     ]
    }
   ],
   "source": [
    "print('Hello')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 保存模型\n",
    "cPickle.dump(lg_goss,open(\"Otto/lg_goss_model.pkl\",'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.412891</td>\n",
       "      <td>0.058633</td>\n",
       "      <td>0.841374</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.068521</td>\n",
       "      <td>0.085005</td>\n",
       "      <td>0.485557</td>\n",
       "      <td>0.516251</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.144047</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.06132</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.064034</td>\n",
       "      <td>0.627034</td>\n",
       "      <td>0.048611</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078015</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.070429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.046683</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.135478</td>\n",
       "      <td>0.033424</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.048236</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.045428</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.063757</td>\n",
       "      <td>0.079708</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.547803</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 94 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0   1      0.000000      0.000000      0.000000      0.000000           0.0   \n",
       "1   2      0.068521      0.085005      0.485557      0.516251           0.0   \n",
       "2   3      0.000000      0.064034      0.627034      0.048611           0.0   \n",
       "3   4      0.000000      0.000000      0.000000      0.046683           0.0   \n",
       "4   5      0.048236      0.000000      0.000000      0.045428           0.0   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  ...  feat_84_tfidf  \\\n",
       "0           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "1           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "2           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "3           0.0      0.000000      0.000000           0.0  ...            0.0   \n",
       "4           0.0      0.063757      0.079708           0.0  ...            0.0   \n",
       "\n",
       "   feat_85_tfidf  feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  feat_89_tfidf  \\\n",
       "0       0.000000       0.412891       0.058633       0.841374       0.000000   \n",
       "1       0.000000       0.000000       0.000000       0.000000       0.144047   \n",
       "2       0.000000       0.000000       0.000000       0.078015       0.000000   \n",
       "3       0.135478       0.033424       0.000000       0.000000       0.000000   \n",
       "4       0.000000       0.000000       0.000000       0.000000       0.000000   \n",
       "\n",
       "   feat_90_tfidf  feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0            0.0       0.000000        0.00000       0.000000  \n",
       "1            0.0       0.000000        0.06132       0.000000  \n",
       "2            0.0       0.000000        0.00000       0.070429  \n",
       "3            0.0       0.000000        0.00000       0.000000  \n",
       "4            0.0       0.547803        0.00000       0.000000  \n",
       "\n",
       "[5 rows x 94 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 进行测试\n",
    "test = pd.read_csv('Otto/Otto_FE_test_tfidf.csv')\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feat_1_tfidf</th>\n",
       "      <th>feat_2_tfidf</th>\n",
       "      <th>feat_3_tfidf</th>\n",
       "      <th>feat_4_tfidf</th>\n",
       "      <th>feat_5_tfidf</th>\n",
       "      <th>feat_6_tfidf</th>\n",
       "      <th>feat_7_tfidf</th>\n",
       "      <th>feat_8_tfidf</th>\n",
       "      <th>feat_9_tfidf</th>\n",
       "      <th>feat_10_tfidf</th>\n",
       "      <th>...</th>\n",
       "      <th>feat_84_tfidf</th>\n",
       "      <th>feat_85_tfidf</th>\n",
       "      <th>feat_86_tfidf</th>\n",
       "      <th>feat_87_tfidf</th>\n",
       "      <th>feat_88_tfidf</th>\n",
       "      <th>feat_89_tfidf</th>\n",
       "      <th>feat_90_tfidf</th>\n",
       "      <th>feat_91_tfidf</th>\n",
       "      <th>feat_92_tfidf</th>\n",
       "      <th>feat_93_tfidf</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.188847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.412891</td>\n",
       "      <td>0.058633</td>\n",
       "      <td>0.841374</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.068521</td>\n",
       "      <td>0.085005</td>\n",
       "      <td>0.485557</td>\n",
       "      <td>0.516251</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.144047</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.06132</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.064034</td>\n",
       "      <td>0.627034</td>\n",
       "      <td>0.048611</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078015</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.070429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.046683</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.135478</td>\n",
       "      <td>0.033424</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.048236</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.045428</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.063757</td>\n",
       "      <td>0.079708</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.163639</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.547803</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 93 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   feat_1_tfidf  feat_2_tfidf  feat_3_tfidf  feat_4_tfidf  feat_5_tfidf  \\\n",
       "0      0.000000      0.000000      0.000000      0.000000           0.0   \n",
       "1      0.068521      0.085005      0.485557      0.516251           0.0   \n",
       "2      0.000000      0.064034      0.627034      0.048611           0.0   \n",
       "3      0.000000      0.000000      0.000000      0.046683           0.0   \n",
       "4      0.048236      0.000000      0.000000      0.045428           0.0   \n",
       "\n",
       "   feat_6_tfidf  feat_7_tfidf  feat_8_tfidf  feat_9_tfidf  feat_10_tfidf  ...  \\\n",
       "0           0.0      0.000000      0.000000           0.0       0.188847  ...   \n",
       "1           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "2           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "3           0.0      0.000000      0.000000           0.0       0.000000  ...   \n",
       "4           0.0      0.063757      0.079708           0.0       0.163639  ...   \n",
       "\n",
       "   feat_84_tfidf  feat_85_tfidf  feat_86_tfidf  feat_87_tfidf  feat_88_tfidf  \\\n",
       "0            0.0       0.000000       0.412891       0.058633       0.841374   \n",
       "1            0.0       0.000000       0.000000       0.000000       0.000000   \n",
       "2            0.0       0.000000       0.000000       0.000000       0.078015   \n",
       "3            0.0       0.135478       0.033424       0.000000       0.000000   \n",
       "4            0.0       0.000000       0.000000       0.000000       0.000000   \n",
       "\n",
       "   feat_89_tfidf  feat_90_tfidf  feat_91_tfidf  feat_92_tfidf  feat_93_tfidf  \n",
       "0       0.000000            0.0       0.000000        0.00000       0.000000  \n",
       "1       0.144047            0.0       0.000000        0.06132       0.000000  \n",
       "2       0.000000            0.0       0.000000        0.00000       0.070429  \n",
       "3       0.000000            0.0       0.000000        0.00000       0.000000  \n",
       "4       0.000000            0.0       0.547803        0.00000       0.000000  \n",
       "\n",
       "[5 rows x 93 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_id = test['id']\n",
    "x_test = test.drop(['id'],axis =1)\n",
    "x_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#保存特征名字后边备用\n",
    "feat_names = x_test.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 稀疏存储，模型训练更快\n",
    "from scipy.sparse import csr_matrix\n",
    "x_test = csr_matrix(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(144368, 9)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = cPickle.load(open('Otto/lg_goss_model.pkl','rb'))\n",
    "y_test_pred = model.predict_proba(x_test)\n",
    "y_test_pred.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>Class_1</th>\n",
       "      <th>Class_2</th>\n",
       "      <th>Class_3</th>\n",
       "      <th>Class_4</th>\n",
       "      <th>Class_5</th>\n",
       "      <th>Class_6</th>\n",
       "      <th>Class_7</th>\n",
       "      <th>Class_8</th>\n",
       "      <th>Class_9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000782</td>\n",
       "      <td>0.126019</td>\n",
       "      <td>0.171635</td>\n",
       "      <td>0.685487</td>\n",
       "      <td>1.399949e-05</td>\n",
       "      <td>0.001791</td>\n",
       "      <td>0.012833</td>\n",
       "      <td>0.000972</td>\n",
       "      <td>0.000467</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.004676</td>\n",
       "      <td>0.006025</td>\n",
       "      <td>0.002514</td>\n",
       "      <td>0.000891</td>\n",
       "      <td>1.729006e-04</td>\n",
       "      <td>0.911126</td>\n",
       "      <td>0.001919</td>\n",
       "      <td>0.067222</td>\n",
       "      <td>0.005454</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000039</td>\n",
       "      <td>0.000091</td>\n",
       "      <td>0.000033</td>\n",
       "      <td>0.000022</td>\n",
       "      <td>5.684073e-07</td>\n",
       "      <td>0.999183</td>\n",
       "      <td>0.000066</td>\n",
       "      <td>0.000488</td>\n",
       "      <td>0.000077</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>0.001200</td>\n",
       "      <td>0.492265</td>\n",
       "      <td>0.479016</td>\n",
       "      <td>0.024757</td>\n",
       "      <td>1.754754e-05</td>\n",
       "      <td>0.000346</td>\n",
       "      <td>0.000639</td>\n",
       "      <td>0.000730</td>\n",
       "      <td>0.001029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.135530</td>\n",
       "      <td>0.002670</td>\n",
       "      <td>0.002315</td>\n",
       "      <td>0.000571</td>\n",
       "      <td>4.234409e-05</td>\n",
       "      <td>0.017620</td>\n",
       "      <td>0.006490</td>\n",
       "      <td>0.073458</td>\n",
       "      <td>0.761302</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id   Class_1   Class_2   Class_3   Class_4       Class_5   Class_6  \\\n",
       "0   1  0.000782  0.126019  0.171635  0.685487  1.399949e-05  0.001791   \n",
       "1   2  0.004676  0.006025  0.002514  0.000891  1.729006e-04  0.911126   \n",
       "2   3  0.000039  0.000091  0.000033  0.000022  5.684073e-07  0.999183   \n",
       "3   4  0.001200  0.492265  0.479016  0.024757  1.754754e-05  0.000346   \n",
       "4   5  0.135530  0.002670  0.002315  0.000571  4.234409e-05  0.017620   \n",
       "\n",
       "    Class_7   Class_8   Class_9  \n",
       "0  0.012833  0.000972  0.000467  \n",
       "1  0.001919  0.067222  0.005454  \n",
       "2  0.000066  0.000488  0.000077  \n",
       "3  0.000639  0.000730  0.001029  \n",
       "4  0.006490  0.073458  0.761302  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out_df = pd.DataFrame(y_test_pred)\n",
    "columns = np.empty(9,dtype=object)\n",
    "for i in range(9):\n",
    "    columns[i] = 'Class_'+str(i+1)\n",
    "out_df.columns = columns\n",
    "out_df = pd.concat([test_id,out_df],axis = 1)\n",
    "out_df.to_csv('Otto/LightGBM_goss_tfidf.csv',index=False)\n",
    "out_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "提交kaggle得分0.46931"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
